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USER’S  GUIDE  FOR  TACTICAL  THINKING 
BEHAVIORALLY  ANCHORED  RATING  SCALES 


I.  Overview 

The  Tactical  Thinking  Behaviorally  Anchored  Rating  Scales  (T-BARS)  are  designed  to 
measure  an  individual’s  cognitive  proficiency  in  tactical  thinking.  By  cognitive  proficiency,  we 
mean  an  individual’s  ability  to  size  up  a  situation,  make  judgments,  and/or  decide  on  courses  of 
action.  T-BARS  were  intended  for  primary  use  in  training  contexts,  but  they  may  be  of  value  in 
other  settings  as  well.  The  scales  are  most  amenable  to  training  sessions  that  require  individuals 
to  respond  to  a  tactical  situation,  such  as  training  vignettes,  tactical  decision  games,  desktop 
simulations,  and  staff  or  field  exercises.  T-BARS  enable  the  evaluator  to  assess  cognition  by 
observing  and  rating  behaviors. 

Phillips,  Shafer,  Ross,  Cox,  and  Shadrick  (2005)  describe  the  process  by  which  the  T- 
BARS  were  developed.  This  User  Guide  is  intended  to  support  the  application  of  T-BARS  for 
assessing  cognitive  proficiency  in  the  tactical  thinking  domain. 

Boundary  Conditions 

The  following  boundary  conditions  apply: 

■  Level.  T-BARS  is  intended  to  measure  individual  performance.  It  is  not  intended  to 
measure  team  or  unit  performance. 

■  Domain.  T-BARS  is  intended  for  use  with  military  personnel  in  the  combat 
arms/ground  combat  domains.  It  is  not  intended  for  any  other  domains. 

■  Rater  qualifications.  T-BARS  is  intended  for  use  by  researchers  or  training 
professionals  who  at  a  minimum  have  an  intermediate  understanding  of  the  nature  of 
applied  cognition  and  are  familiar  with  the  combat  arms  domain.  It  is  also  intended 
for  military  instructors  who  at  a  minimum  have  extensive  knowledge  of  tactical 
decision  making  and  have  experience  assessing  or  examining  learners’  thought 
processes  during  tactical  exercises. 

■  Use  settings.  T-BARS  is  intended  for  scenario-based,  experiential  training  sessions 
requiring  sensemaking  and/or  decision  making,  including  paper-and-pencil  vignette 
training;  tactical  decision  game  or  decision  making  exercise  sessions;  electronic 
vignette  or  decision  making  exercise  sessions;  sandtable  exercises;  desktop 
simulations  or  game-based  trainers;  commander  and  staff  planning  or  execution 
exercises;  field  exercises;  tactical  exercises  without  troops;  or  live  fire  exercises.  It 
can  also  be  used  in  the  context  of  advanced  technology  experimentation  where 
participants  (e.g.,  combat  leaders)  attempt  to  apply  the  technology  in  the  context  of  a 
tactical  problem.  These  settings  may  include  examination  of  net-centric,  battle 
command  technologies  or  examination  of  battlefield  visualization  technologies.  T- 
BARS  is  intended  for  use  in  tactical  situations  with  a  focus  on  either  planning  or 
execution.  It  is  not  intended  for  classroom  lecture  sessions  or  for  training  sessions 
focusing  on  execution  of  specific  tactics  or  procedures — for  example,  practice 
implementing  a  four-man  stack  or  a  standard  room  clearing  operation.  It  is  also  not 
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intended  for  training  for  physical  rather  than  thinking  skills  (e.g.,  weapons  usage 
training). 

■  Nature  of  data  to  be  rated.  T-BARS  is  intended  for  verbal  protocol  data  that  includes 
the  individual’s  reasoning  behind  decisions  and  judgments;  written  responses  to 
tactical  exercises,  especially  courses  of  action  and  accompanying  rationale;  and 
observations  during  tactical  planning  or  execution  exercises.  It  is  not  intended  for 
multiple  choice  or  short  answer  test  data. 

Organization  of  the  User  Guide 


The  remainder  of  this  User  Guide  is  organized  as  follows: 

Section  II  describes  the  theoretical  foundations  of  the  T-BARS  by  discussing  the  role  of 
mental  models  in  cognitive  performance  and  the  manner  by  which  the  T-BARS  attempt  to 
measure  mental  model  maturity  in  the  tactical  thinking  domain. 

Section  III  provides  instructions  for  using  the  T-BARS  in  assessment,  from  collecting 
suitable  data  to  code  with  T-BARS  to  scoring  the  ratings  to  produce  a  cognitive  profile  for  the 
individual  who  has  been  assessed. 

Section  IV,  Establishing  Interrater  Consensus,  provides  recommendations  for  establishing 
consensus  between  raters  using  T-BARS  in  the  same  or  similar  settings. 

Section  V  contains  the  T-BARS  assessment  tool. 
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II.  The  Role  of  Mental  Models 


The  T-BARS  assessment  tool  is  organized  according  to  a  set  of  tactical  mental  models. 
Mental  models  are  internal  representations  of  the  external  world.  They  are  the  mental 
representations  that  encode  an  individual’s  unique  knowledge  gained  from  experience.  Mental 
models  support  understanding,  reasoning,  prediction,  and  action.  Mental  models  are  domain 
specific — they  encapsulate  how  things  work  in  a  particular  domain.  For  leaders  in  combat  arms 
domains,  four  categories  distinguish  the  mental  models  that  are  thought  to  be  primarily 
responsible  for  performance1.  These  four  categories  form  the  structure  of  the  T-BARS: 

*  Know  and  Use  All  Assets  Available  (Assets).  Combat  leaders  must  maintain 
awareness  of  the  synergistic  effects  of  fighting  their  command  as  a  combined  arms 
team.  This  includes  not  only  all  assets  under  their  command,  but  also  those  which 
higher  headquarters  might  bring  to  bear  to  assist  them. 

■  Focus  on  the  Mission  and  Higher’s  Intent  (Mission).  Combat  leaders  must  stay 
aware  of  the  higher  purpose  and  the  results  they  are  directed  to  achieve.  Even  when 
unusual  and  critical  events  may  draw  them  in  a  different  direction,  it  is  essential  to 
stay  focused  on  the  overall  mission.  ..  . 

■  Model  a  Thinking  Enemy  (Enemy).  Combat  leaders  must  remember  that  the 
adversary  is  a  reasoning  human  being  who  is  intent  on  defeating  friendly  forces. 
Although  it  is  tempting  to  simplify  the  battlefield  by  treating  the  enemy  as  static  or 
simply  reactive,  this  will  harm  the  troops’  ability  to  fight  an  effective  battle. 

*  Consider  Effects  of  Terrain  (Terrain).  Combat  leaders  must  not  lose  sight  of  the 
operational  effects  of  the  terrain  on  which  they  must  fight.  Every  combination  of 
terrain  and  weather  has  a  significant  effect  on  what  can  and  should  be  done  to 
accomplish  the  mission. 

» 

T-BARS  is  structured  around  these  four  mental  models  because  performance  on 
cognitive  tasks  (such  as  sensemaking  and  decision  making)  depends  on  the  accuracy  and 
maturity  of  one’s  mental  models.  In  scoring  an  individual’s  developmental  stage  for  each  of  the 
mental  models,  we  have  a  metric  of  overall  cognitive  proficiency. 

General  Stages  of  Mental  Model  Development 

Prior  to  implementing  T-BARS,  it  is  critical  to  understand  the  constructs  of  the  scales.  A 
five-point  scale  has  been  developed  for  each  of  the  aforementioned  mental  models.  The  five 
points  on  each  scale  represent  five  levels  or  stages  of  proficiency,  based  on  the  Dreyfus  & 
Dreyfus  (1986)  model  of  cognitive  skill  acquisition.  The  cognitive  profiles  for  each  level, 
including  characteristics  of  knowledge  and  performance,  are  as  follows: 


1  The  four  mental  models  for  tactical  thinking  were  identified  in  the  Think  Like  A  Commander  program  of  research 
and  training.  For  more  information,  see  Lussier  (1998),  Lussier,  Shadrick,  and  Prevou  (2003),  Phillips,  Shafer,  Cox, 
Ross,  &  Shadrick  (2005),  and  Ross,  Phillips,  Klein,  &  Cohn  (2005). 
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Stage  1:  Novice 


Individuals  who  perform  at  the  Novice  level  have  limited  or  no  experience  in  situations 
characteristic  of  their  domain.  They  are  typically  taught  about  the  situations  they  will  encounter 
in  terms  of  objective  “attributes”  such  as  the  number  of  Soldiers  in  a  unit,  the  range  radius  of 
enemy  assets,  or  other  measurable  quantities  that  can  be  recognized  without  operational  or 
exercise  experience.  Novices  are  also  taught  context-free  rules,  such  as  the  formula  for 
determining  how  long  it  will  take  personnel  carriers  to  get  from  point  A  to  point  B  under  normal 
conditions.  Because  the  Novice’s  understanding  of  the  domain  is  based  largely  in  rules,  his  or 
her  performance  is  quite  limited  and  inflexible.  As  the  study  of  nursing  by  Benner  (1984)  points 
out,  rule-guided  behavior  actually  prevents  successful  performance  because  a  set  of  rules  cannot 
make  clear  which  tasks  are  relevant  or  critical  in  an  actual  situation. 

A  Novice  under  the  Dreyfus  and  Dreyfus  model  may  have  a  great  deal  of  textbook  or 
classroom  knowledge  of  the  domain,  but  what  places  him  or  her  in  Stage  1  is  the  shortage  of 
actual  lived  experience.  There  is  a  clear  distinction  between  the  level  of  performance  that  results 
when  textbook  principles  and  theories  are  applied  and  the  superior  performance  achieved  when 
an  experience  base  guides  performance.  Table  1  summarizes  the  characteristics  of  Novices. 


Table  1 

Stage  1 :  Novice 


STAGE  1:  NOVICE 

General  Characteristics 

Knowledge 

Performance 

•  Objective  facts  and  features  of  the  domain 
(Dreyfus  &  Dreyfus,  1986). 

•  Context-free  (abstract)  rules  to  guide  behavior 
(Dreyfus  &  Dreyfus,  1986). 

■  Domain  characteristics  acquired  through 
textbooks  and  classroom  instruction  (Benner, 
1984). 

*  Guided  by  rules;  is  limited  and  inflexible 
(Benner,  1984). 

■  Shows  recognition  of  elements  of  the  situation 
without  considering  context  (Dreyfus  & 

Dreyfus,  1986). 

■  Is  variable  and  awkward  (Glaser,  1996). 

•  Focuses  on  isolated  variables  (Glaser,  1996). 

■  Consists  of  a  set  of  individual  acts  rather  than 
an  integrated  strategy  (Glaser,  1996;  McElroy 
de  Chesnay  ,  &  Greiner,  1991). 

■  Is  self-assessed  based  on  how  well  he  adheres 
to  learned  rules  (Benner,  1 984;  Dreyfus  & 
Dreyfus,  1986). 

■  Reflects  a  sense  of  being  overwhelmed  since 
all  stimuli  are  perceived  to  be  equally  relevant 
(McElroy  et  al.,  1991). 

Stage  2:  Advanced  Beginner 

Advanced  Beginners  have  acquired  enough  domain  experience  that  their  performance  can 
be  considered  marginally  acceptable.  At  this  stage,  learners  can  recognize,  either  on  their  own  or 
when  pointed  out  to  them  by  an  instructor,  recurring  meaningful  “aspects”  of  the  situation. 
Aspects  are  global  characteristics  that  are  identifiable  only  through  prior  experience;  the  prior 
experience  serves  as  a  comparison  case  for  the  current  situation.  For  example,  an  Advanced 
Beginner  would  be  able  to  grasp  that  close  air  support  could  be  helpful  in  a  particular  situation 
after  taking  part  in  a  previous  exercise  in  which  close  air  support  was  utilized.  A  learner  at  this 
stage  would  not  know  how,  when,  or  where  to  employ  the  air  assets  to  the  best  advantage,  but 
would  recognize  their  potential  to  help  alleviate  the  situation. 

While  it  is  possible  to  make  some  of  these  aspects  explicit  for  a  domain,  it  is  not  possible 
to  form  objective  rules  to  govern  every  situation.  Building  on  the  close  air  support  example,  it  is 
likely  that  a  different  array  of  factors  would  determine  the  applicability  of  air  assets  for  different 
situations.  A  single  set  of  well-defined  rules  cannot  adequately  address  every  instance.  With 
experience,  the  learner  will  increasingly  pick  up  on  the  array  of  cues  that  signal  opportunities  for 
air  support. 

Advanced  Beginners  have  a  set  of  attributes  and  aspects  in  their  repertoire  with  which  to 
guide  their  performance.  At  this  stage  they  can  begin  to  develop  their  own  “guidelines”  that  stem 
from  an  understanding  of  the  domain  attributes  and  aspects.  Guidelines  are  rules  of  thumb  that 
inform  behavior  by  allowing  the  practitioner  to  attach  meaning  to  elements  of  a  situation 
(Dreyfus  &  Dreyfus,  1980;  1986).  For  example,  a  platoon  leader  at  this  level  of  proficiency  may 
know  that  the  first  step  in  conducting  an  offensive  is  to  set  up  a  base  of  fire,  and  he  may  know 
that  the  support  position  should  be  a  certain  distance  from  the  primary  objective.  However,  he 
may  not  understand  that  he  needs  to  take  into  account  not  only  distance  from  the  objective,  but 
also  angles  of  fire,  to  prevent  fratricide.  And  he  probably  cannot  distinguish  that  the  rules  and 
factors  critical  under  one  set  of  circumstances  are  not  necessarily  decisive  in  other  operational 
situations.  Spiro  et  al.  (1992)  note  that  in  complex  domains,  the  application  of  different  patterns 
of  principles  varies  from  situation  to  situation,  and  there  is  substantial  interconnectedness  among 
principles.  At  this  stage  the  practitioner  has  organized  his  or  her  knowledge  and  experience  into 
principles,  but  has  not  built  the  interconnectedness  or  developed  the  ability  for  flexible 
application.  Table  2  summarizes  the  characteristics  of  Advanced  Beginners. 

Stage  3:  Competent 

Stage  3  is  marked  by  the  ability  to  formulate,  prioritize,  and  manage  longer-term  goals  or 
objectives.  This  perspective  gives  the  operator  a  better  sense  of  the  relative  importance  of  the 
attributes  and  aspects  of  the  situation.  The  transition  from  Advanced  Beginner  to  Competent  is 
highlighted  by  a  shift  from  highly  reactive  behaviors,  where  actions  are  taken  right  when  a 
problem  surfaces,  to  more  planned  behaviors,  where  the  learner  can  see  the  larger  picture  and 
assess  what  actions  must  be  taken  immediately  and  what  can  wait  until  later.  While  a  learner  at 
Stage  3  is  not  as  quick  or  flexible  as  a  Stage  4  learner,  he  or  she  can  typically  manage  a  large  set 
of  incoming  information  and  task  demands. 
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Table  2 

Stage  2:  Advanced  Beginner 


ST  A  GE  2:  ADVANCED  BEGINNER 

General  Characteristics 

Knowledge 

Performance 

*  Some  domain  experience  (Benner,  1984; 
Dreyfus  &  Dreyfus,  1986). 

■  More  objective,  context-free  facts  than  the 
novice,  and  more  sophisticated  rules  (Dreyfus 
&  Dreyfus,  1986). 

■  Situational  elements,  which  are  recurring, 
meaningful  elements  of  a  situation  based  on 
prior  experience  (Dreyfus  &  Dreyfus,  1986). 

■  A  set  of  self-generated  guidelines  that  dictate 
behavior  in  the  domain  (Benner,  1984). 

■  Seeks  guidance  on  task  performance  from 
context-rich  sources  (e.g.,  experienced  people, 
documentation  of  past  situations)  rather  than 
rule  bases  (e.g.,  textbooks)  (Houldsworth, 
O’Brien,  Butler,  &  Edwards,  1997). 

■  Is  marginally  acceptable  (Benner,  1984). 

■  Combines  the  use  of  objective,  or  context-free, 
facts  with  situational  elements  (Dreyfus  & 
Dreyfus,  1986). 

■  Ignores  the  differential  importance  of  aspects  of 
the  situation;  situation  is  a  myriad  of  competing 
tasks,  all  with  same  priority  (Benner,  1984; 
Dreyfus  &  Dreyfus,  1986;  Shanteau,  1992). 

■  Shows  initial  signs  of  being  able  to  perceive 
meaningful  patterns  of  information  in  the 
operational  environment  (Benner,  1 984). 

*  Reflects  attitude  that  answers  are  to  be  found 
from  an  external  source  (Houldsworth  et  ah, 
1997). 

■  Reflects  a  lack  of  commitment  or  sense  of 
involvement  (McElroy  et  al.,  1991). 

The  Competent  performer  acts  on  the  situation  with  a  very  analytical,  hierarchical 
approach.  Dreyfus  and  Dreyfus  (1986)  compare  this  to  the  problem-solving  approach  described 
by  proponents  of  information  processing.  Based  on  an  initial  judgment  of  what  part  of  the 
situation  is  most  important,  the  performer  generates  a  plan  to  organize  and  thus  simplify  the 
situation  to  improve  his  performance.  However,  the  drawback  for  Competent  performers  is  that 
their  plans  drive  their  behavior  to  a  greater  extent  than  any  situational  elements  that  may  arise; 
they  tend  to  hesitate  to  change  their  plan  mid-course,  despite  the  introduction  of  new,  conflicting 
information.  Simultaneously,  Competent  performers  are  more  emotionally  invested  in  their 
performance  than  Novices  or  Advanced  Beginners.  Because  they  actively  choose  a  plan  of  action 
for  themselves  rather  than  relying  on  rules  offered  by  a  textbook  or  instructor,  they  take  great 
pride  in  success  and  are  distressed  by  failure.  Table  3  summarizes  the  characteristics  of 
Competent  individuals. 
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Table  3 

Stage  3:  Competent 


STAGE  3:  COMPETENT 

General  Characteristics 

Knowledge 

Performance 

■  How  to  think  about  the  situation  in  terms  of 
overarching  goals  or  tasks  (Benner,  1984). 

■  The  relative  importance  of  subtasks  depending 
on  situational  demands  (Benner,  1984;  Dreyfus 
&  Dreyfus,  1986). 

■  Particular  patterns  of  cues  suggest  particular 
conclusions,  decisions,  or  expectations  (Dreyfus 
&  Dreyfus,  1986). 

■  A  personalized  set  of  guiding  principles  based 
on  experience  (Houldsworth  et  al.,  1997). 

■  How  to  anticipate  future  problems 
(Houldsworth  etal.,  1997). 

•  Is  analytic,  conscious,  and  deliberate  (Benner, 
1984;  Dreyfus  &  Dreyfus,  1986). 

■  Does  not  rely  on  a  set  of  rules  (Houldsworth  et 
al.,  1997). 

■  Is  efficient  and  organized  (Benner,  1984; 

Dreyfus  &  Dreyfus,  1986). 

■  Is  driven  by  an  organizing  plan  that  is  generated 
at  the  outset  of  the  situation  (Dreyfus  &  Dreyfus, 
1986). 

■  Reflects  an  inability  to  digress  from  the  plan, 
even  when  faced  with  new,  conflicting 
information  (Dreyfus  &  Dreyfus,  1986). 

■  Reflects  an  inability  to  see  newly  relevant  cues 
due  to  the  organizing  plan  or  structure  that 
directs  attention  (Benner,  2004). 

■  Reflects  an  emotionally  involved  performer  who 
takes  ownership  of  successes  and  failures 
(Dreyfus  &  Dreyfus,  1986). 

■  Focuses  on  independent  features  of  the  situation 
rather  than  a  synthesis  of  the  whole 
(Houldsworth  et  al.,  1997). 

Stage  4:  Proficient 

Learners  at  the  Proficient  level  have  moved  away  from  perceiving  situations  in  terms  of 
independent  aspects  and  attributes  and  see  the  situation  as  an  inseparable  whole  where  aspects 
and  attributes  are  interrelated  and  woven  together.  The  situation  is  not  deliberately  analyzed  for 
its  meaning;  an  assessment  occurs  automatically  and  dynamically  because  the  learner  has  an 
extensive  experience  base  from  which  to  draw  comparisons.  However,  decisions  regarding 
appropriate  actions  continue  to  require  some  degree  of  detached  analysis  and  deliberation.  With 
regard  to  the  situation  assessment  process,  Proficient  individuals  experience  the  event  from  a 
specific  perspective  with  past  experiences  in  mind.  Therefore,  certain  features  of  the  situation 
stand  out  as  salient,  and  others  fade  into  the  background  as  non-critical.  This  is  in  keeping  with 
findings  from  the  naturalistic  decision  making  research  that  as  individuals  gain  more  experience, 
they  are  quickly  able  to  recognize  critical  cues  and  patterns  of  cues  (e.g.,  Crandall  &  Getchell- 
Reiter,  1993;  Hoffman,  Crandall,  &  Shadbolt,  1998;  Klein,  1998).  Dreyfus  and  Dreyfus  (1986) 
assert  that  at  this  stage,  performers  are  also  positively  impacted  by  new  information  that  is 
obtained  as  the  situation  progresses.  While  Competent  performers  generally  cannot  change  their 
plans  when  faced  with  conflicting  information,  Proficient  individuals  fluidly  adjust  their  plans, 
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expectations,  and  judgments  as  features  of  the  situation  change.  They  have  an  intuitive  ability  to 
recognize  meaningful  patterns  of  cues  without  breaking  them  down  into  their  component  parts 
for  analysis.  Dreyfus  terms  this  ability  “holistic  similarity  recognition.”  However,  the  elements 
that  are  holistically  recognized  must  still  be  assessed  and  combined  using  sophisticated  rules  in 
order  to  produce  a  decision  or  action  that  meets  the  individual’s  goal(s). 

Dreyfus  and  Dreyfus  (1986)  further  describe  Stage  4  performers  as  being  guided  by 
“maxims”  which  reflect  the  nuances  of  a  situation  (see  also  Benner,  1984).  These  maxims  can 
mean  one  thing  under  one  set  of  circumstances,  but  something  else  under  another  set  of 
circumstances.  As  a  simplistic  example,  consider  a  building  in  the  midst  of  an  urban  combat  area 
whose  windows  are  broken  out.  This  cue  could  indicate  that  the  building  is  run  down  and  vacant. 
It  could  also  indicate  that  the  adversary  is  occupying  the  building,  and  has  broken  out  the 
windows  to  use  it  as  a  base  of  fire.  Other  situational  cues  and  factors  will  need  to  be  considered 
to  determine  how  to  interpret  the  broken  out  windows — for  example,  the  adversary’s  history  of 
breaking  out  windows,  typical  building  types  that  he  has  utilized  in  the  past,  his  last  known 
location  and  projected  current  location,  the  presence  or  absence  of  undisturbed  dust  or  dirt 
around  the  building,  and  so  forth. 

While  Proficient  individuals  can  make  note  of  the  maxims  or  nuances,  they  may  not  yet 
be  able  to  reliably  use  the  maxim  to  decide  on  an  action.  Following  the  example  of  the  broken 
out  windows,  a  Proficient  platoon  leader  may  know  to  be  concerned  about  the  enemy’s  presence 
in  the  building,  but  may  lack  the  experience  base  to  integrate  the  situational  cues  and  factors  and 
then  accurately  and  confidently  assess  what  is  actually  happening  in  the  building.  As  a  result,  he 
may  be  unable  to  adjust  his  plan  to  effectively  handle  the  building.  Table  4  summarizes  the 
characteristics  of  Proficient  performers. 

Table  4 


Stage  4:  Proficient 


STAGE  4:  PROFICIENT 

General  Characteristics 

Knowledge 

Performance 

■  Typical  “scripts”  for  categories  of  situations 
(Klein,  1998). 

■  How  to  set  expectancies  and  notice  when  they 
are  violated  (Benner,  1984). 

■  How  to  spot  the  most  salient  aspects  of  the 
situation  (Benner,  1984;  Dreyfus  &  Dreyfus, 
1986). 

■  Personalized  maxims,  or  nuances  of  situations, 
that  require  a  different  approach  depending  on 
the  specific  situation,  but  not  how  to  apply  the 
maxims  correctly  (Benner,  1984;  Houldsworth 
eta!.,  1997). 

■  Reflects  a  perception  of  the  situation  as  a 
whole  rather  than  its  component  features 
(Benner,  1984). 

■  Is  quick  and  flexible  (Benner,  1984). 

■  Reflects  a  focus  on  long-term  goals  and 
objectives  for  the  situation  (Benner,  1984). 

■  Utilizes  prior  experience  (or  intuition)  to  assess 
the  situation,  but  analysis  and  deliberation  to 
determine  a  course  of  action  (Dreyfus  & 

Dreyfus,  1986;  McElroy  et  al.,  1991). 

•  Reflects  a  synthesis  of  the  meaning  of 
information  over  time  (Benner,  2004). 

■  Reflects  a  more  refined  sense  of  timing 
(Benner,  2004). 
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Stage  5:  Expert 


The  fifth  and  final  stage  of  the  Dreyfus  and  Dreyfus  (1986)  model  is  Expertise.  At  this 
level,  the  individual  no  longer  relies  on  analytic  rules,  guidelines,  or  maxims;  performance 
becomes  intuitive  and  automatic.  The  Expert  immediately  understands  which  aspects  of  the 
situation  are  critical  and  does  not  waste  time  on  the  less  significant  aspects.  He  or  she  knows 
implicitly  what  action  to  take  and  can  remedy  a  situation  quickly  and  efficiently.  Table  5 
summarizes  the  characteristics  of  Experts. 

Table  5 

Stage  5:  Expert 


STAGE  5:  EXPERT 

General  Characteristics 

Knowledge 

Performance 

•  How  to  make  fine  discriminations  between 
similar  environmental  cues  (Klein,  1993). 

■  How  to  intuitively  assess  the  situation 
(Benner,  2004;  Dreyfus  &  Dreyfus,  1986). 

■  How  to  respond  to  maxims  or  nuances  based 
on  the  unique  array  of  cues  and  factors  in  the 
situation  (Benner,  2004). 

*  How  to  intuitively  respond  to  the  situation 
(Benner,  1984;  Dreyfus  &  Dreyfus,  1986). 

■  How  tasks  and  subtasks  are  supposed  to  be 
performed  (Phillips,  Klein,  &  Sieck,  2004). 

■  How  equipment  and  resources  function  in  the 
domain  (Phillips  et  al.,  2004). 

■  How  to  perceive  meaningful  patterns  in  large 
and  complex  sets  of  information  (Klein, 

1998;  Dreyfus  &  Dreyfus,  1986). 

■  What  is  typical  and  atypical  for  a  particular 
situation  (Dreyfus  &  Dreyfus,  1986; 

Feltovich,  Johnson,  Moller,  &  Swanson, 

1984;  Klein,  1999). 

■  A  wide  range  of  routines  or  tactics  for  getting 
things  done  (Klein,  1999). 

■  More  facts  about  the  domain  than  less 
proficient  individuals  (Phillips  et  al.,  2004). 

■  A  huge  library  of  lived,  distinguishable 
experiences  that  impact  the  handling  of  new 
situations  (Dreyfus  &  Dreyfus,  1986). 

■  How  to  set  expectancies  and  notice  when  they 
are  violated  (Benner,  1984). 

■  Is  fluid  and  seamless,  like  walking  or  talking; 
“integrated  rapid  response”  (Benner,  1984,  2004; 
Dreyfus  &  Dreyfus,  1986). 

■  Is  based  on  prior  experience  for  both  assessment 
and  decision  making  (Dreyfus  &  Dreyfus,  1986). 

■  Is  automatic,  and  the  rationale  for  actions  is  often 
difficult  to  articulate  (Benner,  1984). 

■  Relies  heavily  and  successfully  on  mental 
simulation  to  predict  events,  diagnose  prior 
occurrences,  and  assess  courses  of  action 
(Einhom,  1980;  Klein  &  Crandall,  1995). 

■  Consists  of  more  time  assessing  the  situation  and 
less  time  deliberating  a  course  of  action  (Lipshitz 
&  Ben  Shaul,  1997). 

■  Shows  an  ability  to  detect  problems  and  spot 
anomalies  early  (Feltovich  et  al.,  1984). 

■  Capitalizes  on  leverage  points,  or  unique  ways  of 
utilizing  ordinary  resources  (Klein  &  Wolf, 

1998). 

■  Reflects  use  of  innovations  and  new  possibilities 
for  responding  to  particular  situations  (like 
leverage  points)  (Benner,  2004). 

■  Manages  uncertainty  with  relative  ease  by  filling 
gaps  with  rational  assumptions  and  formulating 
information-seeking  strategies  (Klein,  1998; 
Serfaty,  MacMillan,  Entin,  &  Entin,  1997). 

■  Reflects  metacognitive  skill,  or  the  ability  to 
self-monitor  (Chi,  1978;  Chi,  Feltovich,  & 

Glaser,  1980;  Larkin,  1983;  Simon,  1975). 

■  Shows  efficient  information  search  activities 
(Shanteau,  1992). 
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Progression  of  Tactical  Thinking  Mental  Model  Development 

Within  each  of  the  four  mental  models,  individuals  begin  their  careers  at  Level  1  as  they 
learn  rules  and  procedures  in  classroom  settings.  As  they  have  the  opportunity  to  apply  the  rules 
they  have  learned  in  training  exercises  (and  possibly  real-world  operations)  they  gradually 
progress  up  the  scale.  As  they  continue  to  experience  an  increasingly  greater  set  of  operational 
circumstances  that  give  them  the  opportunity  to  practice  making  decisions  and  judgments,  they 
move  toward  the  high  end  of  the  scales.  Actual  combat  experience  will  typically  enable  an 
individual  to  progress  to  Levels  4  and  5  much  more  quickly  than  he  or  she  would  otherwise. 

Looking  across  the  four  mental  models,  it  is  hypothesized  that  the  mental  models  follow 
a  particular  developmental  order.  The  rows  in  Figure  1  show  the  four  mental  models  with  the 
descending  order  representing  the  progression  of  their  development.  Knowledge  of  friendly 
assets  (Assets  mental  model)  and  a  focus  on  the  mission  (Mission  mental  model)  seem  to  develop 
first.  As  individuals  enter  the  domain,  they  work  to  grasp  the  “tools”  they  have  to  influence 
situations  and  determine  how  they  can  apply  the  assets  to  tasks.  Once  learners  have  a  basic 
understanding  of  the  assets  at  their  disposal  and  common  mission  tasks  and  objectives,  they  start 
to  consider  the  enemy  as  an  active  component  of  their  planning  and  execution  (Enemy  mental 
model).  In  other  words,  it  appears  to  be  prohibitively  difficult  to  consider  the  role  of  an  opponent 
or  adversary  before  comprehending  ones  own  function  on  the  battlefield.  Finally,  we  suspect 
individuals  initiate  development  of  their  Terrain  mental  model  once  they  have  a  foundation  in 
the  first  three.  Terrain  represents  physical  features  of  the  environment  in  which  the  learner 
applies  his  own  assets,  conducts  mission  tasks,  and  imagines  the  enemy’s  moves.  Meaning 
cannot  be  attributed  to  terrain  features  without  the  context  provided  by  the  former  three  mental 
models,  and  mental  models  do  not  develop  without  contextualized  meaning. 

Note  that  learners  do  not  have  to  master  a  particular  mental  model  before  the  next  one 
can  begin  to  take  shape.  Rather,  a  foundation  must  exist  within  the  early  developing  mental 
models  before  the  later  ones  can  form.  For  example,  we  have  seen  that  individuals  who  lack  a 
sense  of  how  to  employ  their  own  assets  rarely  have  the  ability  to  comprehend  how  terrain 
features  impact  mobility,  prevent  lines  of  sight,  or  present  danger  areas  like  choke  points.  What 
is  not  clear  is  the  extent  to  which  a  foundation  must  be  developed  within  the  early  developing 
mental  models  before  the  later  ones  can  begin  to  develop. 

Furthermore,  four  additional  aspects  of  tactical  thinking  are  hypothesized  to  develop 
when  combat  leaders  are  in  the  advanced  stages  of  tactical  mental  model  development.  These 
additional  components  include  the  following: 

■  See  the  Big  Picture  (Big  Picture).  This  theme  refers  to  the  importance  of 
maintaining  awareness  of  what  is  happening  in  the  environment  and  how  it  might 
affect  operations — what  courses  of  action  can  affect  others’  operations.  A  narrow 
focus  on  one’s  own  fight  can  get  you  blind-sided. 

■  Consider  Timing  (Timing).  The  focus  of  this  theme  is  on  the  importance  of  being 
cognizant  of  the  time  available  to  get  things  done.  A  good  sense  of  how  much  time  it 
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takes  to  accomplish  various  battlefield  tasks  and  the  proper  use  of  that  sense  is  a  vital 
combat  multiplier. 


■  Consider  Contingencies  and  Remain  Flexible  (Contingencies).  Combat  leaders 
must  never  lose  sight  of  the  old  maxim  that  “no  plan  survives  first  contact  with  the 
enemy.”  Flexible  plans  and  well  thought  out  contingencies  result  in  rapid,  effective 
responses  under  fire.  Contingencies  are  characterized  by  thinking  that  begins  with 
questions  like  “What  if...?”  or  “How  else  can  I...?” 


■  Visualize  the  Battlefield  (Visualize).  Leaders  must  be  able  to  visualize  a  fluid  and 
dynamic  battlefield  with  some  accuracy  and  use  this  visualization  to  their  advantage. 
A  leader  who  develops  this  difficult  skill  can  reason  proactively  like  no  other. 

These  four  components  are  seen  as  cognitive  processes  or  mental  manipulations  of  the 
four  mental  models.  They  are  exhibited  by  experienced,  proficient,  combat  decision  makers. 
Leaders  conduct  these  higher  order  mental  operations  in  the  context  of  the  four  basic  mental 
models.  For  example,  an  experienced  tactician  can  estimate  how  long  it  will  take  to  move  a 
bridging  asset  from  one  point  to  another  ( Timing  in  the  context  of  Assets)  or  predict  what  the 
enemy  will  attempt  as  the  situation  plays  out  ( Visualization  in  the  context  of  Enemy). 
Accordingly,  the  columns  in  Figure  1  represent  development  of  skill  in  these  four  cognitive 
processes  around  Levels  4  and  5  of  mental  model  development. 


^  Formation_  of  MentaJ_  Modelj;  ^ 

W  Know  And  Use  Assets 

Know  Data  Unk  Systems  Dynamic 

About  Systems  &  Mission  F"e"d * 

Model  Ai-if-':' 

See  Own  Unit  command 

In  Context  of  The  Force  A 

Larger  Org 

Focus  on  Mission  &  Higher's  Intent 

Focus  on  Own  Discriminate  Model  Effect  of  Own 
Mission  Intent  &  Mission '.'.Mission  on  Higher 

’Headquarters 


Model  a  Thinking  Enemy 

Ignore  Enemy  Use  Enemy  Model  a  Thinking 

Templates  Enemy 


Accurately 

Predict  Impact  Support  Intent 

of  Own  Actions 


Accurately 
Predict  Enemy 
>.v  Actions 


Deny  Euc 
intent 


Consider  Effects  of  Terrain 

OCOKA  Recognize  Dynamic 

Knowledge  Terrain 

Aspects  Model 


See  Uses  of  Shape  the 

Terrain  to  Own  Battlefield  f  • 

Advantage  > 


iSlMSJ 


Figure  1.  Hypothesized  developmental  sequence  of  tactical  mental  models  and  cognitive 
processes. 
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Cognitive  Profiles  for  the  Tactical  Thinking  Mental  Models 

The  following  section  outlines  the  cognitive  profiles  for  each  level  of  the  four  tactical 
thinking  mental  models. 

Know  and  Use  All  Assets  Available 

Level  1 :  Knows  textbook  capabilities.  Performance  is  abstract  and  rule-based,  and 
focuses  on  variables  in  isolation.  Individual  knows  facts  about  standard  capabilities  of  organic 
assets  such  as  ranges  of  weapons,  number  of  vehicles  per  unit,  and  so  forth.  The  foundational 
knowledge  required  to  analyze  how  assets  can  be  applied  to  the  situation  has  not  yet  developed. 

Level  2:  Matches  assets  to  mission  requirements.  Performance  reflects  simple 
analytical  processing  using  a  limited  experience  base.  Organic  assets  are  matched  to  mission 
requirements.  For  example,  a  tank  formation  would  be  allocated  to  the  area  where  heavy  armor  is 
needed  for  protection.  Individual  has  difficulty  prioritizing  tasks,  so  asset  utilization  is  driven  by 
capabilities  (what  the  asset  can  do)  over  situational  demand  (what  is  the  most  pressing  mission 
task). 

Level  3:  Utilizes  organic  assets  to  accomplish  mission  objectives.  Performance  reflects 
a  mental  model  of  asset  utilization,  but  remains  dependent  on  analysis  and  planning  rather  than 
recognition  and  intuition.  Individual  can  prioritize  mission  tasks  and  predict  how  the  situation 
could  unfold,  and  an  asset  utilization  plan  is  generated  against  that  analysis.  However,  execution 
is  driven  by  the  plan  over  the  situation,  so  the  individual  has  difficulty  adjusting  asset  utilization 
to  meet  changing  situational  demands. 

Level  4:  Recognizes  full  range  of  assets  required  based  on  situational  demands. 

Performance  reflects  a  recognitional  or  intuitive  assessment  of  the  situation,  but  analytical 
decision  making  where  the  individual  deliberates  about  a  course  of  action.  Individual  recognizes 
the  availability  of  non-organic  and  non-military  assets  in  addition  to  his  own  organic  assets.  For 
example,  civilians  are  recognized  to  be  valuable  sources  of  human  intelligence  (HUMINT). 
Situational  demands  drive  asset  utilization,  rather  than  the  plan  or  the  organic  assets  at  the 
individual’s  disposal. 

Level  5:  Applies  full  range  of  assets  to  direct  the  outcome  of  the  battle.  Performance 
reflects  a  recognitional  ability  to  assess  and  decide.  Individual  can  visualize  specific  outcomes  of 
asset  utilization  and  has  the  ability  to  avoid  unwanted  consequences.  For  example,  he  knows  how 
to  command  and  maneuver  his  forces  to  avoid  an  uprising  by  the  locals.  Individual  leverages  and 
coordinates  organic,  non-organic,  and  non-military  assets  to  achieve  mission  objectives. 

Focus  on  the  Mission  and  Higher ’s  Intent 

Level  1:  Focuses  on  own  mission.  Performance  is  abstract  and  rule-based,  and  focuses 
on  variables  in  isolation.  Individual  fixates  on  own  mission  rather  than  considering  larger 
organization’s  mission.  He  is  unable  to  consider  higher  intent.  The  foundational  knowledge 
required  to  analyze  steps  necessary  for  mission  accomplishment  has  not  yet  developed. 
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Level  2:  Discriminates  intent  and  explicit  mission.  Performance  reflects  simple 
analytical  processing  using  a  limited  experience  base.  Mission  tasks  are  paramount  to  all  else, 
and  intent  can  be  articulated  but  not  operationalized.  Individual  has  difficulty  prioritizing  tasks 
for  mission  accomplishment  and  is  often  uncertain  or  overwhelmed  as  the  situation  evolves. 
There  is  a  tendency  to  rely  on  direction  from  higher  headquarters  (HQ)  rather  than  making  own 
decisions. 

Level  3:  Models  effects  of  own  mission  and  HQ  intent.  Performance  reflects  a  mental 
model  of  how  intent  is  achieved  through  mission  tasks,  but  remains  dependent  on  analysis  and 
planning  rather  than  recognition  and  intuition.  Individual  can  prioritize  mission  tasks  and  predict 
how  the  situation  could  unfold,  and  a  course  of  action  is  generated  based  on  that  analysis. 
However,  performance  in  execution  is  guided  by  an  efficient  but  rigid  plan  that  is  not  adapted  to 
account  for  changes  in  the  situation. 

Level  4:  Makes  accurate  predictions.  Performance  reflects  a  recognitional  or  intuitive 
assessment  of  the  situation,  but  analytical  decision  making  where  the  individual  deliberates  about 
a  course  of  action.  Individual  recognizes  how  situational  factors  impact  the  mission  and  the  path 
to  achieving  intent.  For  example,  he  can  visualize  the  enemy’s  likely  objective  and  use  of  terrain, 
and  he  uses  that  assessment  to  deliberate  about  how  to  support  the  intent  through  his  own 
mission.  During  execution,  changes  in  the  situation  are  recognized  intuitively,  and  mission  tasks 
are  adapted  or  changed  to  continue  to  support  intent. 

Level  5:  Supports  intent.  Performance  reflects  a  recognitional  ability  to  assess  and 
decide.  Individual  can  quickly  and  accurately  assess  the  situation,  visualize  contingencies,  and 
devise  an  action  plan  that  accomplishes  the  intent  while  avoiding  unwanted  2nd  and  3rd  order 
consequences.  Individual  operates  from  a  big  picture  perspective  in  which  he  takes  actions  that 
support  the  short-  and  long-term  objectives  of  the  coalition  force. 

Model  a  Thinking  Enemy 

Level  1:  Uses  enemy  templates.  Performance  is  abstract  and  rule-based.  Individual 
acknowledges  enemy  superficially  and  equates  him  with  theoretical  or  doctrinal  templates 
learned  in  schoolhouse — for  example,  the  typical  Soviet  offensive  formation.  The  foundational 
knowledge  required  to  analyze  probable  enemy  actions  and  objectives  has  not  yet  developed. 

Level  2:  Regards  enemy  as  static.  Performance  reflects  simple  analytical  processing 
using  a  limited  experience  base.  Enemy  is  understood  to  have  an  impact  on  the  mission,  but  is 
regarded  as  a  static,  non-thinking  adversary.  Individual  has  trouble  distinguishing  enemy  centers 
of  gravity  from  the  rest  of  the  enemy  picture.  Individual  struggles  to  make  sense  of  or  draw 
hypotheses  about  the  enemy’s  objectives. 

Level  3:  Regards  enemy  as  intelligent  and  dynamic.  Performance  reflects  a  mental 
model  of  an  intelligent,  dynamic  enemy.  Individual  analyzes  the  enemy  situation  and  predicts 
enemy  actions  in  order  to  formulate  an  efficient  and  organized  course  of  action  (COA)  that 
defeats  the  enemy.  Ideas  about  the  enemy’s  objectives  and  COA  are  constructed,  but  they  are 
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general  and  imprecise.  Because  the  individual  is  guided  by  the  plan  rather  than  situational 
demands,  he  struggles  to  adapt  his  COA  when  the  enemy  situation  changes  during  execution. 

Level  4:  Predicts  enemy  actions.  Performance  reflects  a  recognitional  or  intuitive 
assessment  of  the  enemy’s  objective  and  intent,  but  analytical  decision  making  where  the 
individual  deliberates  about  a  course  of  action  that  defeats  the  enemy.  Individual  continually 
updates  his  assessment  of  the  enemy  situation  and  his  predictions  about  the  enemy’s  next  steps 
based  on  situational  factors. 

Level  5:  Denies  enemy  intent.  Performance  reflects  a  recognitional  ability  to  assess  and 
decide.  Individual  visualizes  how  enemy  will  act  and  react,  and  takes  actions  to  deny  enemy 
intent.  For  example,  he  recognizes  enemy  leverage  points  and  takes  action  to  neutralize  them  or 
make  them  unavailable. 

Consider  Effects  of  Terrain 

Level  1:  Uses  terrain  checklists.  Performance  focuses  on  identifying  discrete  features  of 
terrain.  Individual  uses  standard  checklists  to  determine  relevant  terrain  features.  The 
foundational  knowledge  required  to  analyze  the  impact  of  terrain  on  the  mission  has  not  yet 
developed. 

Level  2:  Identifies  important  terrain  features.  Performance  reflects  simple  analytical 
processing  using  a  limited  experience  base.  Important  terrain  features  are  identified  and 
prominent  problem  areas  such  as  chokepoints  are  avoided.  However,  individual  remains  unable 
to  leverage  terrain  to  own  advantage. 

Level  3:  Incorporates  terrain  into  own  plan.  Performance  reflects  a  mental  model  of 
the  impact  of  terrain  on  the  mission.  Individual  performs  an  analysis  of  the  terrain  and 
incorporates  terrain  features  into  the  plan.  For  example,  in  an  urban  setting  the  tallest  and 
sturdiest  buildings  are  perceived  as  good  locations  to  occupy.  However,  the  individual  tends  to 
adhere  to  the  plan  even  after  the  situation  has  evolved  and  new  information  about  the  terrain 
becomes  available. 

Level  4:  Recognizes  how  the  enemy  may  use  terrain.  Performance  reflects  a 
recognitional  or  intuitive  assessment  of  the  aspects  and  patterns  of  terrain  that  are  critical  for 
friendly  and  enemy  forces  but  deliberate  analysis  of  how  to  utilize  the  terrain  to  accomplish  the 
mission.  Individual  continually  updates  his  view  of  the  terrain  and  its  impact  on  the  mission  as 
the  situation  evolves  and  new  terrain  features  and  patterns  are  discovered. 

Level  5:  Turns  terrain  to  own  advantage.  Performance  reflects  a  recognitional  ability 
to  assess  and  decide.  Individual  is  quickly  able  to  visualize  how  terrain  will  impact  the  friendly 
mission  and  predicted  enemy  actions.  He  leverages  the  terrain  to  his  own  advantage  and  denies 
the  enemy’s  ability  to  do  the  same. 
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Using  the  T-BARS  Assessment  Tool 

The  T-BARS  assessment  tool  will  enable  you  to  judge  the  current  state  of  an  individual’s 
tactical  mental  models.  It  provides  a  means  to  measure  improvement  over  time  and  as  a  result  of 
training  interventions.  An  individual’s  profile  and  score  has  implications  for  appropriate  next 
steps  in  his/her  training  cycle.  When  measured  in  the  context  of  examining  battle  command 
technologies,  an  individual’s  profile  can  provide  an  understanding  of  what  the  system  contributes 
to  his  or  her  cognitive  process. 

To  conduct  an  assessment  with  T-BARS,  it  is  critical  to  have  a  thorough  data  record  with 
which  to  score  behaviors.  We  have  identified  three  sources  of  performance  data:  verbal 
protocols,  written  responses,  and  observations. 

Utilization  of  a  verbal  protocol  is  likely  the  best  approach  for  capturing  a  thorough  record 
of  performance  for  an  individual.  In  contrast  to  using  written  responses,  a  researcher 
implementing  a  verbal  protocol  has  more  control  to  ensure  that  questions  are  answered  as 
intended.  Similarly,  when  contrasted  with  observations,  a  verbal  protocol  enables  the  researcher 
to  pose  specific  questions  about  the  participant’s  thought  process  rather  than  relying  solely  on 
what  the  participant  communicates  as  part  of  the  exercise.  A  verbal  protocol  also  requires  less 
work  from  the  learner  or  participant  than  do  written  responses.  For  these  reasons  we  advocate 
verbal  protocol  data  as  the  basis  for  scoring  performance.  However,  this  approach  is  admittedly 
more  time  consuming  and  expensive  than  others.  We  therefore  provide  recommendations  for 
collecting  and  scoring  data  from  each  of  the  three  sources. 

Verbal  Protocol  Data 


Examples  of  Settings 

Verbal  protocol  data  may  be  captured  as  part  of  a  training  exercise  or  as  a  stand-alone 
session.  In  any  case,  it  is  necessary  to  base  the  session  around  a  scenario  or  tactical  situation  to 
which  the  participant  must  respond.  The  scenario  might  be  a  TLAC  vignette,  a  tactical  decision 
game,  or  even  a  simulation  of  a  tactical  incident. 

Data  Collection  Recommendations 

The  goal  should  be  to  capture  the  participant’s  decisions,  judgments,  and  the  thought 
process  and  rationale  behind  them.  To  do  so,  ask  the  following  types  of  questions: 

■  What  are  the  critical  issues  or  facts  to  consider? 

■  What  is  your  assessment  of  the  situation? 

■  How  could  the  situation  unfold  with  what  you  know  now? 

■  What  would  be  your  biggest  concern  at  this  point?  What  would  be  your  top  priority? 

■  How  would  you  approach  this  situation?  What  would  you  do? 

■  Have  you  seen  any  situation  like  this  before?  Briefly  describe. 

■  What  other  information  do  you  want  about  this  situation,  if  any?  How  would  you  get 
[element  of  information  mentioned]? 

■  How  would  that  information  affect  your  understanding  of  the  situation? 
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You  will  need  to  customize  your  protocol  to  the  vignette  you  are  conducting.  For 
example,  when  new  information  becomes  available  in  the  scenario  or  when  the  situation  has 
changed,  you  might  ask  a  question  about  how  the  change  impacts  the  participant’s  situation 
assessment.  Or,  if  a  subordinate  in  the  exercise  provides  a  situation  report  or  asks  for  direction, 
you  might  ask  a  question  about  how  the  participant  will  respond  to  the  subordinate.  In  Appendix 
A,  we  provide  an  example  of  a  vignette  script  and  the  verbal  protocol  questions  posed  to  the 
participant  during  the  vignette  session. 

Additional  tips  for  conducting  the  verbal  protocol  include: 

■  Conduct  the  session  one-on-one.  You  should  be  eliciting  responses  from  one 
individual  at  a  time. 

■  Take  time-outs  during  the  session.  By  pausing  the  exercise  or  tactical  problem  at 
points  where  new  information  has  been  provided  to  the  participant,  you  can  capture 
his/her  thought  process  as  he/she  works  through  the  implications  of  the  additional 
information. 

■  Preface  the  verbal  protocol  session  by  informing  the  participant  that  you  would  like  to 
hear  everything  that  he  or  she  is  thinking  about.  Encourage  the  participant  to  tell  you 
as  much  as  possible  as  he  or  she  answers  the  protocol  questions,  even  if  he  or  she 
believes  the  thought  to  be  obvious  or  a  “no-brainer.” 

■  Follow  up  on  answers  by  asking  “Why?” 

■  Follow  up  on  answers  by  asking  “Is  there  anything  else  you’re  thinking  about?”  to 
ensure  that  you  are  getting  as  much  information  about  the  participant’s  thoughts  and 
considerations  as  possible. 

Written  Response  Data 


Examples  of  Settings 

As  with  the  verbal  protocol  approach,  written  responses  must  be  captured  in  the  context 
of  a  tactical  scenario  or  simulation.  For  example,  you  may  present  participants  with  a  paper- 
based  tactical  decision  making  exercise  or  a  computer-based  tactical  situation  that  evolves  over 
time. 

Data  Collection  Recommendations 

The  goal  should  be  to  capture  the  participant’s  decisions,  judgments,  assessments,  and  the 
thought  process  and  rationale  behind  them.  To  do  so,  develop  questionnaires  with  the  following 
types  of  questions: 

■  What  is  your  assessment  of  the  situation? 

■  What  is  your  plan? 

*  What  orders  will  you  give  your  subordinates? 

■  Describe  the  rationale  behind  your  plan. 

■  What  is  your  greatest  concern? 

■  What  do  you  believe  will  happen  as  the  situation  evolves,  and  why? 


16 


Additional  tips  for  eliciting  good  written  response  data  include: 

■  Provide  a  great  deal  of  white  space  in  which  participants  can  respond  to  each 
question.  Extensive  space  to  write  conveys  the  expectation  that  the  answers  should  be 
in  the  form  of  paragraphs  rather  than  one  or  two  sentences. 

■  Give  participants  plenty  of  time  to  write  their  responses.  Encourage  them  to  write  as 
much  as  possible. 

■  To  ensure  thorough  responses,  keep  the  participant’s  mental  and  work  load 
reasonably  low.  Responses  written  at  the  end  of  a  four-hour  training  session  will  not 
be  as  complete  as  those  written  after  only  one  or  two  hours  of  training.  We  also 
recommend  eliciting  responses  for  no  more  than  two  tactical  situations  in  a  single 
session.  When  participants  have  to  answer  the  same  questions  again  and  again,  their 
responses  gradually  lose  their  thoroughness. 

■  If  the  simulation  enables  the  participant  to  act  on  the  tactical  situation  and  change  it 
(e.g.,  a  game-based  trainer),  consider  taking  time-outs  in  the  simulation  during  which 
the  participant  can  write  responses  to  the  questions.  This  ensures  that  he  or  she  is 
reporting  judgments  and  assessments  in  the  moment  rather  than  attempting  to 
remember  at  the  end  of  the  session  what  he  or  she  was  considering  earlier  in  the 
simulation. 

■  If  the  session  takes  place  with  multiple  participants  (e.g.,  a  decision  making  exercise 
in  a  classroom  setting),  have  participants  complete  their  response  forms  prior  to 
engaging  in  discussion  or  after-action  review  with  other  participants  or  an  instructor. 

Observational  Data 


Examples  of  Settings 

Observations  may  be  conducted  during  group  training  exercises  to  assess  the  cognitive 
performance  of  individuals.  Typically,  a  group  leader  will  be  the  target  for  the  assessment.  One 
example  of  a  suitable  setting  is  a  planning  exercise  for  a  commander  and  his  staff.  As  with  the 
other  two  data  types,  observational  data  must  also  be  collected  in  the  context  of  a  tactical 
scenario  or  problem. 

Data  Collection  Recommendations 

Observations  will  be  more  difficult  to  code  using  T-BARS  than  the  other  two  data  types 
due  to  the  cognitive  demand  placed  on  the  rater.  He  or  she  will  have  to  listen  and  watch 
performance  and  quickly  choose  a  behavioral  descriptor  to  account  for  the  performance.  To  do 
this,  the  researcher  must  be  very  familiar  with  the  content  of  the  T-BARS  in  advance.  He  or  she 
must  have  a  solid  understanding  of  the  cognitive  profiles  for  each  level  within  each  theme  as 
well  as  a  recollection  of  the  main  (or  most  often  used)  bulleted  behavioral  descriptors  within 
each  scale. 

The  following  recommendations  apply  to  coding  observational  data: 

■  When  possible,  use  at  least  two  researchers  to  rate  each  participant  being  observed. 
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*  Develop  an  observation  guide  that  enables  you  to  briefly  document  the  behavior  or 
utterance  in  one  column,  and  the  two  T-BARS  ratings  (theme  and  level)  in  two  other 
columns. 

■  To  minimize  the  amount  of  page  turning  while  observing,  we  recommend  that  you 
use  large  (e.g.,  11x17)  paper  and  print  each  of  the  four  T-BARS  on  a  single  sheet. 

■  Expect  that  you  will  not  be  able  to  rate  as  many  behaviors  while  observing  them  as 
you  would  be  able  to  rate  using  verbal  protocol  or  written  response  data  over  the 
same  period  of  participant  performance.  Due  to  the  increased  workload  in  this  setting, 
the  segments  of  data  you  will  code  will  be  larger  (and  thus  fewer)  than  if  you  were 
working  with  a  session  transcript. 

Preparing  Data  for  Coding 


Verbal  Protocol  Data 

To  code  verbal  protocol  data,  you  will  need  to  break  the  data  record  (i.e.,  transcript)  into 
several  data  segments.  Each  segment  should  represent  a  single,  distinctive  behavior  or  thought. 
The  segments  will  be  short.  They  may  be  a  sentence  or  two.  They  may  be  a  portion  of  a  sentence. 
It  is  not  likely  that  a  single  behavior/thought  will  span  more  than  three  sentences. 

The  following  is  an  excerpt  from  a  transcript  where  the  data  has  been  segmented  (“///” 
represents  a  new  segment).  The  bold  type  denotes  researcher  questions  and  comments. 

My  mission  has  not  changed.  I  still  have  to  get  going  forward.  That’s  not  a  problem.  Ill  I’m  now 
starting  to  get  a  little  concerned  about  my  communications  capability.  Platoon  Leader  of  my 
supported  host  nation  feels  the  same  way  -  we  need  to  keep  moving.  Ill  And  getting  them  the  best 
help  right  now  is  not  going  to  be...l  can’t  help  them  at  this  point  in  time.  Ill  1  don’t  have  the 
assets  to  do  that.  And  I’m  not  going  to  cut  my  strength  and  detach  anybody.  Everything  has  to  get 
through.  Ill  The  only  thing  I  can  send  through  the  translator  to  the  village  elder  is  “we  will  be 
back.”  Not  we  will  be  back,  we’re  going  to  have  to  send  back  we  will  notify  people  ahead  of 
time.  Ill  I’ll  also  be  interested  in  finding  out  who  he  talked  to.  He’s  got  (inaudible)  how  to  get  it 
done.  Ill  Is  there  any  other  information  you’d  like  to  have  at  this  point?  From  the  elders  or 
from...?  Ill  Just  about  the  situation.  If  you  had  an  information  wand,  what  three  wishes 
would  you  make?  I  know  he  has  people  he’s  concerned  about  as  an  elder,  getting  his  people  out 
of  there.  That  may  die  as  a  result  of  me  not  digging  them  out.  Ill  I’d  love  to  have  commo  back  to 
the  rear.  I’d  love  to  be  able  to  call  them  right  now.  Ill  Everything  has  to  get  through,  so  I  can’t 
leave  anything,  and  I  can’t  leave  anybody.  So  right  now,  I’ve  got  to  keep  pushing  on. 
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After  you  have  segmented  the  data,  enter  it  into  a  coding  table.  We  suggest  the  following 
format  shown  in  Table  6. 

Table  6 

Sample  Coding  Table  Including  Data  Segments 


I 

I 

I 


Data  Segment 


Things  I  need  to  know.  I’m  thinking  about,  I’ve 
got  to  know  the  condition  of  the  route  between  A 
and  T  and  T  to  Q  and  L. 


I’ve  got  some  minimal  information  regarding 
obstacles  within  the  city. 


I’ve  got  no  information  regarding  where  sniper 
fire  is  coming  from,  where  it  seems  to  be  focused 
and  so  forth.  I’d  want  to  get  more  information  on 
that  if  I  could. 


In  terms  of  the  critical  tasks,  as  I  see  this  in 
sequence,  is  I’ve  got  to  do  a  route  reconnaissance. 
I’m  not  gonna  proceed  down  that  route  until 
reconnaissance  is  complete. 


And  I’ve  got  to  think  about  how  to  task  organize 
that  route  reconnaissance.  I’m  thinking  about  how 
I’m  going  to  organize  my  road  marks  to  provide 
the  best  security  that  I  can.  I’ve  got  to  consider 
possibility  of  ambush,  possibility  of  mines,  having 
to  do  mine  clearing  operations,  and  be  structured 
to  do  that. 


I’ve  got  to  organize  for  movement  to  contact  so  I 
can  develop  the  situation  without  having  my 
whole  force  committed  along  a  route. 


I’ve  got  to  have  more  information,  I  need  more 
structure,  more  detailed  information  about  the 
town  of  Agdam,  the  high  speed  mounted 
approaches,  how  movement  occurs  in  that  city, 
where  the  key  intersections  are  that  facilitate  the 
movement  of  forces  in  and  around  that  city.  That 
will  of  course  then  lead  me  to  determine  how  I’m 
going  to  array  and  deploy  my  forces  in  order  to 

secure  that  site,  which  is  also  my  task. _ 

There’s  also  a  requirement,  for  example,  to  secure 
a  helicopter  landing  area.  That  task  in  itself, 
there’s  specific  terrain  that’s  suitable  for 
helicopter  operations.  I  don’t  have  expertise  in 
that  in  my  organization,  but  personally  I  know 
generally  what  those  requirements  would  be. 


Theme  Level 


In  the  Theme  column  you  will  enter  one  of  the  four  themes  or  mental  models — Assets, 
Mission,  Enemy,  or  Terrain — addressed  by  the  data  segment.  In  the  Level  column  you  will  enter 
a  number  1-5  to  reflect  which  level  of  performance  corresponds  to  the  behavior/thought  from  the 
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data  segment.  The  Notes  column  can  be  used  to  insert  comments  about  the  code  to  yourself  or  to 
the  other  rater(s).  For  example,  you  may  lack  confidence  in  your  rating  due  to  ambiguity  or 
vagueness  in  the  data  segment.  In  this  case  you  may  wish  to  come  back  to  that  segment  once  you 
have  read  more  of  the  transcript  and  better  understand  the  participant’s  meaning.  You  may  also 
wish  to  confer  with  the  other  rater(s)  on  his  or  her  interpretation  of  the  segment. 

During  the  coding  process  you  may  find  that  some  segments  encapsulate  more  than  one 
theme  or  level.  In  these  cases,  you  may  wish  to  further  break  down  the  data  into  additional 
segments. 

Written  Response  Data 

Written  response  data  should  be  handled  in  much  the  same  way  as  verbal  protocol  data. 
Refer  to  the  previous  section  for  guidance  on  preparing  written  response  data  for  coding. 

Observational  Data 

Observation-based  coding  is  unique  in  that  you  will  be  generating  the  data  record  while 
you  are  assigning  ratings.  You  also  will  typically  lack  a  thorough  record  of  performance,  as  you 
will  not  be  able  to  capture  everything  that  is  said  and  done  by  the  participant.  We  recommend  the 
following  format  in  Table  7  to  support  observational  data  coding: 

Table  7 

Sample  Format  for  Coding  Observational  Data 


Utterance  or  Behavior 

Theme 

Level 

While  coding,  use  the  first  column — Utterance  or  Behavior — to  summarize  the  behavior 
you  are  observing  and  coding.  For  example,  “Describes  route  selection”  or  “Identifies  need  to 
communicate  with  local  leader.”  It  is  likely  that  you  will  not  be  able  to  document  the  full  content 
of  the  utterance  or  behavior  (e.g.,  “chooses  Route  Green  because  there  have  been  no  improvised 
explosive  devises  (lEDs)  on  that  road  and  because  it  traverses  fewer  urban  areas  than  Routes 
Blue  or  Yellow”)  due  to  the  need  for  quick  documentation.  Rather,  summarize  the  behavior  and 
let  the  code  you  assign  to  it  characterize  the  content. 
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Sample  Ratings 


Table  8  below  provides  an  example  of  verbal  protocol  data  from  three  separate  interviews 
that  has  been  coded.  It  links  data  segments  to  the  Themes,  Levels,  and  specific  Behavioral 
Descriptors  that  describe  the  cognitive  performance  of  the  participant.  Note  that  it  is  not 
necessary  to  document  specific  behavioral  descriptors  when  implementing  the  T-BARS  tool.  We 
provide  the  descriptors  here  to  indicate  examples  of  how  they  are  operationalized  within  a  data 
record. 

Table  8 

Example  of  Coded  Verbal  Protocol  Data 


# 

Data  Segment 

Theme 

Level 

Behavioral  Descriptor  j 

1 

Look  at  my  road.  I  have  no. .  .I’m 
heading  further  up  into  the  valley, 
so  I’m  not  going  to  be  able  to 
make  (inaudible)  and  I  have  no 
commo  back  to  my  rear.  Which  is 
not  good. 

Assets 

2 

(C)  IDescribes  general  posture  for  organic 
assets  to  take  rather  than  specific  tasks. 

2 

My  mission  has  not  changed.  I 
still  have  to  get  going  forward. 
That’s  not  a  problem. 

Mission 

2 

(C)  States  what  needs  to  be  accomplished 
(mission  task)  but  not  how  to  do  it.  jj 

3 

I’m  now  starting  to  get  a  little 
concerned  about  my 
communications  capability. 

Platoon  leader  of  my  supported 
host  nation  feels  the  same  way  - 
we  need  to  keep  moving. 

Assets 

2 

(C)  Articulates  a  consideration  of  the 

safety  and  security  of  assets  (including 
Soldiers). 

1 

And  getting  them  the  best  help 
right  now  is  not  going  to  be. .  .1 
can’t  help  them  at  this  point  in 
time. 

Mission 

2 

(C)  Makes  a  statement  about  the  situation 
(planning  and  execution)  in  terms  of 
mission  analysis  without  intent  as  a 
“lens.” 

5 

I  don’t  have  the  assets  to  do  that. 
And  I’m  not  going  to  cut  my 
strength  and  detach  anybody. 
Everything  has  to  get  through. 

Assets 

3 

(C)  Identifies  trade-offs,  benefits,  and  risk 
of  splitting  or  reassigning  assets. 

6 

The  only  thing  I  can  send  through 
the  translator  to  the  village  elder 
is  “we  will  be  back.”  Not  we  will 
be  back,  we’re  going  to  have  to 
send  back,  we  will  notify  people 
ahead  of  time. 

Mission 

2 

(C)  Makes  a  statement  about  the  situation 
(planning  and  execution)  in  terms  of 
mission  analysis  without  intent  as  a 
“lens.” 

7 

Is  there  any  other  information 
you’d  like  to  have  at  this  point? 

From  the  elders  or  from. . .? 

Assets 

3 

(C)  Asks  questions  about  availability  of 
non-organic  assets,  [big  picture] 

8 

Just  about  the  situation.  If  you 
had  an  information  wand,  what 

Assets 

4 

(C)  States  consequences  or  effects  of  asset 
usage  beyond  specific  primary  effect. 
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(Table  Continues) 


Data  Segment 

Theme 

Level 

I  know  he  has  people  he’s 
concerned  about  as  an  elder, 
getting  his  people  out  of  there. 

That  may  die  as  a  result  of  me  not 
digging  them  out. 

Behavioral  Descriptor 


1 

Also  we  need  to  know  what 
groups  or  tribes,  political  groups 
are  along  that  route,  what  areas, 
territories  we  may  be  moving 
through  to  address  any  kind  of 
political  issues  that  might  exist 
there 

Terrain 

2 

and  also  gather  further 
intelligence  on  the  potential 
opposing  forces  that  might  be 
there. 

Enemy 

Describes  process  by  which  judgment 
should  be  made  about  terrain. 


(H)  Asks  questions/seeks  information  about 
what  enemy  is  doing  in  own  sector. 


1  think  I  mentioned  intelligence 
but  the  key  thing  is  gathering 
more  information  on,  get  a  quick 
route  recon,  I  would  want  a  plan 
in  more  detail  about 
reconnaissance  with  detailed 
information  about  the 
chokepoints,  the  aspects  of,  what 
we  call  local  control  on  that  route, 
I  mentioned  the  tribes,  the  local 
groups  or  factions,  I’m  trying  to 
define  very  specifically  with  great 
details  about  at  what  road 
intersections  or  buildings  or 
whatever  that  could  cause  a 
change  or  a  group  change  to 
occur. 


Identifies  information  needed  on  terrain, 
for  example,  features  or  conditions  that 
must  be  identified  during  planning  due  to 
implications  for  mission  (e.g.,  mosques, 
nature  of  a  road). 


I’d  want  as  much  intelligence  as  I 
can  possibly  get  on  the 
enemy/potential  enemy  within 

that  whole  map  of  the  city, _ 

How  would  you  expect  that  the 
situation  might  unfold  from 
here?  I  would  expect  light 
resistance  maybe;  I  would  expect 
light  resistance/wouldn’t  expect 
NO  resistance,  but  I  would  expect 
light  resistance. 


Enemy 


Enemy 


(H)  Asks  questions/seeks  information  about 
what  enemy  is  doing  in  own  sector. 


Makes  a  projection  about  how  enemy  or 
populace  will  react  to  own  actions. 
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Frequently  Asked  Questions  for  Data  Coding 

Question:  Should  I  take  into  account  the  context  around  the  data  segment  -  for  example,  the 
preceding  segments  and  the  segments  that  follow  it  -  when  I  code  it? 

If  the  meaning  of  the  data  segment  being  rated  is  clear  without  using  the  surrounding 
context,  then  do  not  use  the  context  around  it  to  make  a  judgment.  In  cases  where  the 
participant’s  utterance  or  behavior  is  unclear  due  to  an  ambiguous  reference  (e.g.,  to  an  unnamed 
person,  location,  route,  etc.),  or  an  unfinished  sentence,  for  example,  you  may  wish  to  use  the 
context  around  it  to  make  a  judgment. 

Question:  What  do  I  do  if  I  don  ’ t  understand  the  meaning  of  the  participant’s  utterance? 

If  even  after  using  the  surrounding  context  you  cannot  understand  what  the  participant 
has  stated,  do  not  code  that  item. 

Question:  What  if  more  than  one  behavioral  descriptor  seems  to  account  for  the  data 
segment? 

Select  the  one  behavioral  descriptor  that  BEST  describes  the  data  segment.  If  more  than 
one  descriptor  from  the  same  level  seems  to  account  for  the  data  segment,  there  is  not  an  issue; 
simply  assign  that  level  to  the  segment.  If  descriptors  from  separate  levels  seem  to  account  for 
the  data  segment,  re-read  the  descriptors  for  subtle  differences  in  their  meaning  and  select  the 
best  description  for  the  data  segment.  If  this  fails,  consider  the  cognitive  profiles  for  each  level  in 
question  and  assign  the  level  that  best  describes  the  individual’s  proficiency  as  indicated  by  the 
data  segment. 

Question:  How  do  /  deal  with  a  participant  who  changes  his  mind  from  one  segment  to  the 
next? 


An  individual  who  flip-flops  back  and  forth  as  to  how  to  handle  a  situation  is 
experiencing  uncertainty  as  to  how  to  interpret  the  situation,  or  as  to  how  to  act  on  his/her 
interpretation  of  the  situation.  Code  each  segment  independently.  When  you  score  the  data  after 
all  the  coding  is  complete,  any  variability  in  level  ratings  will  be  averaged  out. 

Question:  What  do  I  do  if  I  can  7  find  a  behavioral  descriptor  from  the  T-BARS  that  accounts 
for  the  data  segment? 

In  this  case,  consider  the  data  segment  uncodeable  and  move  to  the  next  segment. 
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Scoring  T-BARS  Ratings 


Generating  Scores 

Once  you  have  coded  all  the  data  segments  for  an  individual,  the  next  step  is  to  generate  a 
score  for  him/her  on  each  of  the  four  mental  model  scales  -  Assets,  Mission,  Enemy,  and 
Terrain. 

An  individual’s  score  is  simply  the  average  of  the  level  ratings  for  each  of  the  four  mental 
models.  Thus,  you  will  produce  four  numerical  scores  ranging  from  1 .0  to  5.0. 

First,  group  the  coded  data  segments  according  to  theme/mental  model.  All  the  Assets- 
related  segments  go  together,  all  the  Mission-related  segments  go  together,  and  so  forth.  Then 
calculate  the  average  for  each  theme  by  adding  all  the  level  ratings  and  dividing  by  the  number 
of  segments  within  that  theme.  Round  the  resulting  number  to  the  nearest  tenth.  Put  the  four 
scores  together  to  generate  an  individual  profile  like  that  shown  in  Table  9. 

Table  9 

Example  Score  Profile 


Participant:  CPT  Jones 

Theme 

Score 

Assets 

3.7 

Mission 

3.5 

Enemy 

3.2 

Terrain 

2.9 

In  situations  where  more  than  one  rater  has  coded  data  for  an  individual,  we  recommend 
reconciling  any  discrepancies  in  coding  prior  to  calculating  the  score  profile.  See  Section  IV  on 
Interrater  Reliability  for  tips  on  improving  consensus  across  raters. 

When  raters  disagree  on  theme,  you  may  either  1)  discuss  the  item  and  determine  that  one 
rater  will  change  his/her  theme  rating  to  match  the  other  rater’s  code,  or  2)  count  the  data 
segment  twice  in  the  overall  score  calculation,  once  for  each  theme  to  which  it  has  been 
attributed. 

For  data  segments  on  which  raters  agree  on  theme  but  do  not  agree  on  level,  you  may  1) 
discuss  the  item  and  determine  that  one  rater  wishes  to  change  his/her  level  code  to  match  the 
other  rater’s  code,  or  2)  average  the  two  level  ratings  and  calculate  these  new  averaged  level 
ratings  into  the  overall  score  for  the  theme. 

Making  Sense  of  a  Score  Profile 

The  score  profile  you  generate  for  an  individual  enables  you  to  1)  make  comparisons  with 
other  score  profiles  (e.g.,  when  examining  the  impact  of  a  training  intervention  or  battle 
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command  support  technology  on  leamer/user  cognition);  or  2)  diagnose  his  or  her  level  of 
cognitive  proficiency  and  performance  as  a  tactical  thinker.  When  your  goal  is  the  former,  it  is 
relatively  straightforward  to  make  sense  of  a  score  profile.  You  can  judge  whether  one  or  more 
of  the  tactical  mental  models  improve  or  decline  as  a  result  of  the  training  or  technology  by 
examining  whether  the  scores  increase  or  decrease. 

When  the  goal  of  applying  T-BARS  is  to  diagnose  an  individual’s  cognitive  proficiency, 
the  score  profile  should  do  two  things:  First,  it  should  indicate  whether  a  particular  mental  model 
(or  set  of  mental  models)  is  underdeveloped  relative  to  the  others.  For  example,  if  the  participant 
scores  a  ‘4’  on  Terrain  but  a  ‘2’  on  Enemy,  there  is  an  indication  that  future  training  sessions 
should  focus  on  strengthening  models  of  a  thinking  adversary.  Second,  it  provides  a  general 
indication  of  the  participant’s  overall  stage  of  cognitive  development  across  the  four  mental 
models.  For  individuals  in  combat  arms  domains,  we  expect  mental  model  development  to 
follow  a  general  sequence  illustrated  in  the  Figure  1 .  The  Assets  and  Mission  mental  models  are 
hypothesized  to  begin  developing  first.  The  Enemy  mental  model  will  begin  to  form  once  the 
individual  has  a  foundation  in  Assets  and  Mission.  Finally,  the  Terrain  mental  model  initiates 
development  once  the  other  three  have  matured  to  some  degree.  It  is  not  clear  to  what  extent  the 
early  mental  models  must  develop  before  the  later  ones  can  begin  to  develop.  Once  individuals 
reach  Level  3  or  4,  they  begin  to  manipulate  the  mental  models  and  gain  skills  in  Seeing  the  Big 
Picture,  Considering  Timing,  Considering  Contingencies,  and  Visualizing  the  Battlefield, 
typically  in  that  order. 

With  the  hypothesized  developmental  sequence  in  mind,  you  can  use  the  score  profile 
across  the  four  mental  models  to  approximate  the  learner’s  overall  level  of  proficiency.  For 
example,  a  score  profile  where  Assets  =  3.6,  Mission  =  3.8,  Enemy  =  3.2,  and  Terrain  =  2.7 
indicates  that  the  individual  is  approximately  at  Stage  3,  or  Competent,  overall.  Likewise,  an 
individual  who  scores  mostly  in  the  range  from  3.5  -  4.5  would  be  around  Stage  4,  Proficient, 
overall.  And  a  learner  who  scores  in  the  range  from  1 .5  -  2.5  is  likely  to  be  a  Stage  2  Advanced 
Beginner. 

Note  that  the  practice  of  labeling  individuals  according  to  the  five  stages  of  cognitive 
skill  acquisition  is  not  an  exact  science.  Dreyfus  and  Dreyfus  (1986)  warn  that  the  five  stage 
model  enables  you  to  label  a  particular  performance,  but  not  necessarily  an  individual.  It  is  our 
contention  that  T-BARS  scores  should  be  used  as  general  indicators  of  an  individual’s 
proficiency  on  the  specific  task  or  exercise  that  produced  the  score.  Repeated  application  of  T- 
BARS  to  examine  the  same  learner’s  performance  across  a  range  of  tactical  problems  provides 
an  increasingly  accurate  measure  of  the  individual’s  overall  proficiency.  When  the  goal  is  to 
diagnose  an  individual’s  cognitive  development,  the  real  value  of  T-BARS  and  the  five-stage 
model  is  to  suggest  next  steps  for  training.  The  score  profile  is  not  intended  for  personnel 
selection  or  to  diagnose  an  individual’s  potential  in  any  way. 

Implications  of  a  Score  Profile 

The  paragraphs  below  offer  suggestions  for  training  individuals  according  to  their  current 
approximate  stage  of  cognitive  development.  For  each  stage,  there  are  recommendations  for  the 
content  of  training  scenarios  as  well  as  instructional  strategies  that  are  thought  to  be  best  suited 
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for  facilitating  the  individual’s  movement  toward  the  next  stage.  These  recommendations  are 
taken  from  Ross,  Phillips,  Klein,  and  Cohn  (2005).  Ross  and  her  colleagues  present  these 
guidelines  as  educated  extrapolations  from  the  research  literature  on  cognitive  skill  acquisition, 
but  they  have  not  undergone  formal  validation. 

Novices 

Key  Components  of  Scenario  Design.  Existing  military  training  is  very  strong  for 
individuals  at  the  novice  level.  They  require  standard  rules  to  anchor  their  thinking  and 
knowledge  about  how  to  execute  procedures.  However,  introduction  of  scenarios  into  the 
novice’s  training  program  can  assist  them  in  developing  an  understanding  of  when  and  how  the 
rules  and  procedures  apply  operationally.  Tactical  decision  making  exercises  for  novices  should 
utilize  a  ground-based  (rather  than  bird’s  eye)  perspective  and: 

*  Focus  on  utilization  of  assets  and  requirements  for  mission  accomplishment.  Most  of 
the  content  of  the  scenarios  should  enable  novices  to  practice  executing  procedures 
and  tactics  (e.g.,  movement  to  contact,  establishing  a  blocking  position)  in  context. 
Learners  should  practice  on  a  range  of  scenarios  that  illustrate  how  tactics  must  be 
implemented  somewhat  differently  depending  on  the  particulars  of  the  situation. 
Exercises  should  also  require  learners  to  allocate  assets  to  various  mission  tasks  and 
receive  embedded  feedback  about  the  effective  range  of  weapons  in  context  and  in 
the  time  it  takes  to  traverse  between  points  given  situational  factors  such  as  road 
conditions  and  weather.  They  should  illustrate  for  the  learner  that  a  unit  is  not  a  single 
fused  entity  as  it  appears  on  a  tactical  map,  but  rather  consists  of  moving  pieces  and 
parts  (like  people  and  vehicles).  This  enables  learners  to  begin  forming  mental 
models  of  assets  that  can  be  split  up  or  attached  to  other  units,  and  conceptualizing 
groupings  that  occupy  more  than  a  static  grid  coordinate  on  a  map. 

■  Incorporate  simple  aspects  of  a  dynamic  enemy.  At  this  level  it  is  useful  for  scenarios 
to  exhibit  to  the  learner  that  the  enemy  is  not  static.  He  moves  around  the  battlefield 
and  splits  up  his  forces  (e.g.,  puts  snipers  alone  in  towers)  just  like  friendlies  can  do. 
At  the  novice  level,  this  is  sufficient  introduction  of  the  enemy. 

■  Incorporate  simple  but  meaningful  terrain  features.  Scenarios  can  include  hills  or 
other  elevated  areas  that  impact  line  of  sight.  They  can  introduce  features  that  will 
exhibit  the  difference  between  cover  and  concealment  like  buildings  (cover  and 
concealment,  depending  on  the  construction)  or  automobiles  (concealment  but  not 
cover).  And  they  can  present  dirt  versus  paved  roads  that  differentially  affect  rates  of 
movement. 

Instructional  Strategies.  Novices  will  be  best  served  by  practicing  on  a  range  of  scenarios 
with  different  assets  available  and  different  mission  requirements.  The  goal  is  to  help  them  begin 
to  develop  their  mental  models  across  a  wide  range  of  asset  and  mission  types  in  order  to  better 
understand  asset  capabilities  and  mission  tasks  in  context. 

At  the  novice  level,  instructors,  or  coaches,  or  mentors  are  necessary  to  guide  and  direct 
the  learning  process  more  so  than  at  the  later  stages.  Following  tactical  exercises,  an  instructor- 


26 


led  after-action  review  should  focus  on  the  following  lines  of  questioning  and  probing  regarding 
the  learners’  experiences: 

■  Asset  capabilities.  What  was  learned  about  how  to  use  the  assets’  capabilities  in  the 
context  of  the  situation? 

■  Mission.  What  actions  were  taken  and  why?  Was  the  mission  accomplished?  Why  or 
why  not? 

■  Enemy.  What  was  learned  about  the  enemy?  What  was  surprising  about  the  enemy? 
How  might  learners  think  about  him  differently  next  time? 

■  Terrain.  How  did  terrain  features  impact  the  mission? 

Advanced  Beginners 

Key  Components  of  Scenario  Design.  Advanced  beginners  are  ready  to  make  meaning 
out  of  the  experiences  they  glean  from  scenario-based  training.  At  this  level,  scenarios  can 
supplement  existing  training  by  enabling  learners  to  practice  implementing  tactics  that  have  been 
newly  introduced  and  employing  assets  whose  capabilities  they  are  learning.  In  addition, 
scenarios  for  advanced  beginners  should  incorporate  enemy  and  terrain  models  that  are  more 
complex  than  those  in  the  novice  scenarios.  Specifically: 

■  Scenarios  should  reflect  an  intelligent,  dynamic  adversary.  The  enemy  should  not 
follow  the  templates  that  have  been  taught  in  classroom  instruction  or  case  study 
analysis.  Enemy  forces  should  move  and  take  action  while  the  learner  deliberates 
about  his  own  actions.  The  goal  is  to  break  the  learner  out  of  the  mindset  that  the 
enemy  will  be  predictable  and  static. 

■  Scenarios  should  incorporate  terrain  that  has  significant  impact  on  the  workability  of 
potential  courses  of  action.  For  example,  movement  along  a  straight,  flat  road  should 
result  in  being  spotted  and  engaged  by  the  enemy.  Furthermore,  enemy  courses  of 
action  should  leverage  terrain  features  (e.g.,  pin  friendly  forces  in  a  choke  point  or 
against  an  unfordable  river)  to  illustrate  the  role  of  terrain  on  the  battlefield. 

■  Asset  capabilities  should  continue  to  be  exercised,  just  as  at  the  novice  level. 
Scenarios  should  incorporate  units  that  are  not  full  strength  or  units  that  have  had 
assets  attached  or  detached  (or  lost  in  previous  battles).  At  platoon  level  and  lower, 
friendly  assets  must  be  depicted  as  individual  moving  pieces  (Soldiers,  vehicles,  and 
so  forth)  rather  than  as  unit  icons  that  move  as  a  whole.  Further,  some  scenarios 
should  reward  learners  for  thinking  ahead  about  what  other  assets  might  be  needed 
and  keeping  a  reserve  to  deal  with  future  events.  Other  scenarios  should  reward  the 
decisive  employment  of  the  learner’s  full  force.  Learners  need  to  develop  an 
understanding  of  the  trade-offs  of  keeping  a  reserve  element  or  not,  and  begin  to 
project  ahead  to  assess  what  might  happen  in  the  future  that  will  require  preparation 
and  readiness.  Finally,  scenarios  can  illustrate  how  assets  can  be  used  to  acquire 
information  to  reduce  levels  of  uncertainty.  Learners  should  receive  useful 
information  (e.g.,  about  the  enemy’s  activities  or  other  important  battlefield  features) 
from  assets  that  are  positioned  to  see  a  wider  view  than  the  learner  himself;  in  this 
way,  advanced  beginners  can  develop  mental  models  about  how  to  proactively 
acquire  information. 
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■  The  mission  required  by  the  scenario  should  be  relatively  simple  and  straightforward, 
and  should  correspond  to  tactics  and  missions  that  have  been  taught  in  classroom  or 
analogous  instructional  settings.  However,  some  advanced  beginner  scenarios  should 
incorporate  mission  tasks  that  must  be  prioritized,  such  that  learners  fail  if  they  do  not 
address  the  higher-priority  task  first. 

Instructional  Strategies.  Advanced  beginners  would  benefit  from  practicing  with  the 
same  scenario  several  times.  Multiple  iterations  allow  learners  at  this  level  to  understand  how 
different  uses  of  assets  and  various  courses  of  action  impact  the  outcome.  Instructors  should 
encourage  experimentation  at  this  stage,  even  with  courses  of  action  that  are  judged  to  be  non- 
optimal.  It  is  important  for  learners  to  internalize  the  specific  reasons  that  some  courses  of  action 
produce  better  results  than  others.  And,  learners  may  find  unexpected  positive  outcomes  from  a 
particular  course  of  action.  If  possible,  instructors  should  be  able  to  introduce  small  alterations  in 
the  environmental  conditions  to  illustrate  how  variations  in  situational  factors  influence  the 
workability  and  “goodness”  of  available  courses  of  action. 

Like  novices,  advanced  beginners  still  require  an  instructor  to  guide  and  direct  their 
learning  process.  After-action  reviews  should  be  instructor-led,  and  can  address  the  following 
lines  of  questioning: 

■  Utilization  of  assets.  What  worked,  what  did  not  work,  and  what  factors  need  to  be 
considered  when  deciding  how  to  employ  assets  (e.g.,  morale,  readiness)? 

■  Mission  tasks.  How  were  the  tasks  approached,  and  why?  Which  approaches  were 
beneficial  and  which  were  not?  Why  and  why  not? 

■  Enemy.  What  did  the  enemy  do,  and  why?  How  did  learners  know  what  he  was 
doing?  What  information  led  to  their  assessments?  Were  their  assessments  accurate, 
and  why  or  why  not? 

■  Terrain.  What  features  were  noticed  during  planning?  How  did  terrain  impact  the 
mission  during  execution,  and  why?  How  would  the  learners  approach  the  terrain 
layout  differently  next  time? 

Competent  Individuals 

Key  Components  of  Scenario  Design.  Tactical  exercises  for  competent  performers  should 
enable  continued  development  of  Asset,  Mission,  Enemy,  and  Terrain  mental  models,  but  in  the 
context  of  the  Consider  Timing  and  Consider  Contingencies  cognitive  processes.  In  other  words, 
scenarios  should  present  situations  where  success  relies  on  the  timing  and  sequencing  of  the 
operation,  planning  for  contingencies,  and  adapting  contingency  plans  as  the  mission  progresses. 
Specifically: 

■  Scenarios  should  introduce  surprises  during  the  execution  of  missions  to  provide 
practice  in  rapidly  responding  to  the  changed  situation.  For  example,  friendly  units 
could  become  unable  to  perform  (e.g.,  because  they  cannot  reach  their  intended 
position,  or  because  a  weapon  system  breaks  down);  the  enemy  could  move  in  a  non- 
traditional  way  or  bring  a  larger  force  than  was  reported  by  intelligence  or 
reconnaissance;  key  roads  could  be  too  muddy  to  traverse  or  blocked  by  refugees 
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demanding  assistance;  or,  higher  headquarters  could  deliver  a  new  frag  order  based 
on  an  opportunistic  target  or  other  change  in  the  situation. 

■  Scenarios  should  present  conflicts  that  require  prioritization  of  mission  tasks. 

Learners  need  to  be  forced  to  determine  which  part  of  the  mission  order  is  most 
important  to  higher  headquarters  based  on  the  commander’s  intent  statement.  Success 
should  be  contingent  on  taking  actions  that  support  intent. 

■  Mission  orders  should  incorporate  strict  time  requirements,  and  the  scenarios  should 
build  in  realistic  timing  of  force  movement,  engagement  with  the  enemy,  and  so  forth. 
If  success  relies  on  accomplishing  an  objective  by  a  particular  time,  and  learners  are 
forced  to  make  judgments  about  how  long  the  prerequisite  tasks  will  take,  then  good 
feedback  about  those  timing  judgments  will  be  available. 

■  Scenarios  should  require  proper  sequencing  of  tasks  in  order  for  the  learner  to 
accomplish  the  mission.  That  is,  learners  should  be  able  to  see  how  the  mission 
breaks  down  when  certain  tasks,  like  thorough  route  reconnaissance,  are  not 
accomplished  prior  to  other  tasks,  like  moving  forces  along  a  route. 

■  Scenarios  should  introduce  the  utility  of  non-organic  and  non-military  assets.  For 
example,  learners  can  be  encouraged  to  request  assets  from  higher  headquarters  or 
another  by  realizing  that  the  mission  can  only  be  accomplished  by  accessing  those 
assets.  Also,  scenarios  can  present  civilian  resources  such  as  host  nation  officials, 
village  elders,  relief  workers,  or  U.S.  ambassadorial  staff  members  who  can  provide 
valuable  information  or  serve  important  roles  (like  communicating  with  the  local 
populace). 

Instructional  Strategies.  At  the  competent  level,  instructors  play  a  key  role  in  mental 
model  development,  but  their  participation  at  the  competent,  proficient,  and  expert  levels  is  not 
required  as  persistently  as  it  is  for  novices  and  advanced  beginners.  In  lieu  of  an  instructor, 
feedback  can  be  delivered  by  developing  expert  responses  against  which  learners  can  compare 
their  own  performance.  Also,  feedback  should  illuminate  situational  cues,  factors,  or  demands 
that  should  have  prompted  learners  to  change  their  approach  or  move  to  a  contingency  plan.  The 
following  issues  should  be  addressed  with  individuals  at  the  competent  level: 

■  Prior  to  execution,  contingencies.  What  are  the  different  ways  the  plan  could  play  out, 
and  how  would  the  learner  know  if  that  were  happening? 

■  Prior  to  execution,  the  enemy.  What  might  he  be  attempting  to  do,  and  why?  How 
might  the  learner  assess  the  enemy’s  objectives  as  the  situation  plays  out?  What 
information  should  the  learner  be  seeking? 

■  Prior  to  execution,  terrain.  What  are  the  critical  terrain  features  on  the  battlefield? 
How  might  they  impact  both  friendly  and  enemy  courses  of  action?  How  might 
terrain  be  leveraged  and  used  against  the  enemy?  How  might  the  enemy  leverage 
terrain  features  and  use  them  against  friendlies? 

■  Mission  plan.  Why  did  the  plan  break  down?  What  should  have  been  the  early 
indicators  that  plan  wouldn’t  play  out  as  intended? 

■  Situation.  What  were  the  cues  and  factors  available?  How  might  they  have  been 
interpreted? 

■  Timing  and  sequencing.  What  issues  regarding  timing  and  sequencing  needed  to  be 
considered  and  why? 
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■  The  Big  Picture.  What  was  higher  HQ  trying  to  accomplish?  What  was  the  learner’s 
role  in  accomplishing  the  larger  mission?  Did  the  learner  contribute  in  useful  ways  to 
the  larger  mission? 

Proficient  Individuals 

Key  Components  of  Scenario  Design.  In  general,  tactical  exercises  for  proficient 
individuals  should  incorporate  high  levels  of  complexity,  ambiguity  and  uncertainty, 
sophisticated  coordination  requirements,  and  situations  that  evolve  and  change  rapidly  into  tough 
dilemmas.  More  specifically: 

■  Scenarios  should  present  situations  where  accomplishing  commander’s  intent  requires 
a  different  approach  than  accomplishing  the  explicit  mission  tasks. 

■  Scenarios  should  incorporate  an  enemy  who  uses  non-conventional  forces  and 
techniques.  For  example,  the  enemy  could  use  civilian  vehicles,  dress  deceptively,  or 
otherwise  mislead  the  learner. 

■  Scenarios  should  incorporate  substantial  situational  changes  during  execution  to  force 
the  learner  to  revise  the  existing  course  of  action  or  develop  a  new  one  on  the  fly. 
Proficient  performers  should  be  skilled  at  recognizing  how  the  situation  has  changed, 
but  they  require  multiple  repetitions  in  order  to  develop  and  refine  the  action  scripts 
within  their  mental  models. 

■  Scenarios  should  incorporate  feedback  on  secondary  and  tertiary  consequences  of 
action.  For  example,  in  a  nation  building/humanitarian  assistance  mission,  an 
emotion-driven  decision  to  provide  assistance  to  desperate  villagers  rather  than 
continuing  with  the  original  mission  may  have  consequences  for  mission 
accomplishment  and  domino  into  a  larger  impact  on  the  operation.  Depending  on  the 
particulars  of  the  situation,  an  action  like  this  could  prompt  locals  to  set  unwarranted 
expectations  about  how  relief  is  provided,  or  bog  down  relief  efforts  to  a  greater  need 
elsewhere,  or  have  political  ramifications  for  how  coalition  efforts  are  portrayed, 
especially  if  the  assistance  provided  was  insufficient  due  to  its  pop-up  nature. 

■  Scenarios  should  require  timing,  sequencing,  and  coordination  between  and  across 
units  rather  than  only  within  the  learner’s  own  organic  assets.  This  enables  learners  to 
form  mental  models  of  friendly  forces  as  a  larger  team  effort  and  to  understand  the 
capabilities  and  limitations  of  other,  dissimilar  units  (e.g.,  air  or  artillery). 

Instructional  Strategies.  The  facilitation,  in  whatever  form  it  takes,  should  exhaust  the 
learner’s  way  of  understanding  and  approaching  the  situation.  He  should  be  required  to  cite  his 
own  personal  experiences  and  exemplars  for  perspective  on  his  views  of  the  situation  depicted  in 
the  scenario  (Benner,  2004).  Benner  recommends  that  instructors  teach  inductively,  where  the 
learner  sees  the  situation  and  then  supplies  his  own  way  of  understanding  the  situation. 

When  an  instructor  is  available  to  facilitate  the  scenario-based  training,  semi-structured 
time-outs  during  execution  of  the  scenarios  would  be  beneficial.  These  periods  of  inquiry  and 
reflection  could  encourage  learners  to  discuss  their  current  interpretation  of  the  situation,  their 
mental  simulation  of  how  the  situation  is  likely  to  play  out,  and  their  ideas  about  what  courses  of 
action  are  most  likely  to  produce  the  desired  results.  Discussion  amongst  the  learners  would  be 
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nearly  as  valuable  for  proficient  performers  as  the  probes  and  dialogue  with  the  instructor. 
Likewise,  after-action  reviews  following  execution  of  a  mission  should  encourage  dialogue  and 
questioning  between  the  instructor  and  learners  about  their  interpretations  of  the  situation,  their 
mental  simulations  and  visualizations  of  the  battlefield,  and  especially  their  consideration  of  how 
various  courses  of  action  supported  or  failed  to  support  the  mission  goals. 

When  an  instructor  is  unavailable,  a  couple  of  different  approaches  can  provide  adequate 
substitutes  for  proficient  learners.  First,  as  noted  above,  individuals  at  this  stage  of  development 
can  learn  quite  a  bit  from  their  peers.  Semi-structured  after-action  reviews  can  be  provided  to 
groups  of  learners  to  guide  their  discussion  of  the  exercise.  The  reviews  should  focus  on  the 
same  questions  used  when  an  instructor  is  present — how  the  situation  was  assessed,  how  learners 
projected  into  the  future,  and  the  rationale  for  the  courses  of  action  employed  or  adjustments  that 
would  be  made  to  actions  based  on  the  outcomes  of  the  scenarios.  In  addition,  learners  should  be 
encouraged  to  share  past  experiences  that  have  influenced  their  thinking  about  the  scenario. 

Just  as  competent  learners  are  likely  to  benefit  from  expert  responses,  proficient  learners 
can  also  use  information  generated  from  the  experts  as  another  instructor-free  approach.  At  the 
proficient  level,  expert  responses  should  include  very  detailed  information  about  how  experts 
thought  about  the  scenario  at  multiple  intervals  within  the  scenario,  if  possible.  This  information 
can  be  generated  by  conducting  in-depth,  cognitive  task  analysis  (CTA)-like  interviews  with  a 
group  of  two  to  four  expert  tacticians.  In  particular,  learners  should  be  provided  with  experts’ 
interpretations  of  the  situation,  including  the  cues  and  factors  they  recognized  pertaining  to  the 
enemy  objective,  the  friendly  status,  and  other  aspects  of  the  battlefield  (e.g.,  terrain  or  non- 
combatants).  They  should  also  be  shown  the  experts’  projections  (i.e.,  mental  simulations)  about 
how  the  situation  would  play  out  and  the  rationale  for  those  projections  as  well  as  visualizations 
of  first,  second,  and  third  order  consequences.  There  should  also  be  a  discussion  about  the 
courses  of  action  taken  by  the  experts  along  with  a  detailed  rationale  regarding  asset  allocation, 
prioritization  and  primary  goal(s),  and  aspects  of  timing  and/or  sequencing. 

Other  topics  to  review  following  scenario-based  training  sessions,  with  or  without 
instructor  leadership,  include  the  following: 

■  The  larger  picture.  What  is  the  larger  organization  trying  to  accomplish?  How  can  the 
learner  develop  opportunities  for  the  larger  organization,  or  otherwise  feed  the  overall 
objective  over  and  above  own  mission  tasks? 

■  Enemy  intent.  What  is  his  likely  intent?  What  aspects  of  the  situation  could  have 
revealed  clues  about  his  intent?  How  can  it  (or  was  it)  denied  by  friendly  forces? 

■  Contingencies.  What  are  some  alternative  ways  the  situation  could  have  played  out? 
What  situational  cues  would  suggest  those  particular  outcomes?  What  responses 
(i.e.,  courses  of  action)  would  be  appropriate  for  the  alternative  contingencies? 

■  Actions.  What  courses  of  action  could  be  taken  in  response  to  changes  in  the 
dynamics  of  the  situation?  What  are  the  relative  advantages  and  disadvantages  of 
each? 
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Experts 


Key  Components  of  Scenario  Design.  If  experts  are  involved  in  scenario-based  training 
sessions,  they  may  reap  the  greatest  benefit  as  a  mentor  to  less  experienced  tacticians.  By 
coaching  and  being  forced  to  communicate  what  they  know  to  others,  they  reflect  on  and  thus 
strengthen  their  existing  mental  models.  It  may  also  be  possible  to  develop  Garden  Path 
scenarios  (Feltovich  et  al.,  1984)  in  training  to  challenge  the  fine  discriminations  within  experts’ 
mental  models. 

Instructional  Strategies.  Experts  are  likely  to  benefit  from  peer  discussion  groups  that 
reflect  on  a  shared  real-world  experience,  full-scale  exercise  or  simulation,  or  operational 
planning  session  (e.g.,  plan  development  to  address  a  potential  crisis  situation  in  a  real-world 
“hot  spot;”  experts  are  generally  the  ones  called  in  to  develop  plans  and  contingencies,  and  this 
setting  can  provide  a  learning  opportunity  for  highly-skilled  tacticians).  Discussions  could  be 
structured  to  address: 

■  Enemy  intent.  What  was  (is)  the  enemy’s  objective  and  why?  What  situational  cues 
and  factors  led  to  that  assessment?  At  what  point  in  the  mission  did  the  enemy’s 
intent  and  his  course  of  action  become  clear?  What  were  the  key  indicators? 

■  Big  picture.  How  did  (could)  individual  units,  or  joint/coalition  forces,  work  together 
to  meet  the  overarching  mission?  Were  assets  shared  in  ways  that  supported  mission 
accomplishment?  What  other  configurations  of  assets  could  have  addressed  the  larger 
mission  intent,  rather  than  unit-specific  orders? 

*  Contingencies.  Did  the  mission  play  out  in  unexpected  ways  that  were  not  imagined 
in  contingency  planning  sessions?  When  was  the  change  noticed?  Were  there  early 
indicators  that  could  have  revealed  the  new  direction  to  commanders  sooner? 

■  Visualization.  What  were  (are)  the  friendly  and  enemy  leverage  points  on  the 
battlefield?  How  did  (could)  friendly  forces  deny  enemy  intent  by  using  the  terrain, 
non-conventional  assets  (e.g.,  civilians),  and  other  resources  or  strategies? 
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IV.  Establishing  Interrater  Consensus 


If  you  plan  to  use  T-BARS  within  your  organization  as  a  way  to  assess  tactical  thinking 
proficiency,  you  may  desire  to  use  multiple  raters  to  assess  performance.  In  this  case  you  will 
need  to  establish  interrater  reliability  to  ensure  that  all  raters  are  using  the  scales  in  a  consistent 
manner.2  To  establish  consensus,  we  recommend  the  following  process  to  be  performed  prior  to 
implementation  of  the  T-BARS  to  produce  an  assessment: 

Instructions  for  Establishing  Interrater  Consensus 

1 .  Generate  a  data  set  and  break  it  down  into  data  segments  as  described  in  Section  III. 
We  recommend  25-35  data  segments  to  rate  during  this  process.  Attempt  to  include 
data  from  participants  from  a  range  of  performance  levels  so  that  you  can  practice 
using  all  five  levels  of  the  T-BARS.  Also  attempt  to  include  data  that  will  reflect  all 
four  themes.  For  example,  some  tactical  exercises  do  not  place  much  emphasis  on 
terrain-related  judgments.  Ensure  that  the  tactical  exercise(s)  from  which  you  pull 
data  require  the  participant  to  consider  all  four  themes. 

2.  Generate  a  coding  table  like  the  one  on  page  21  to  be  filled  out  by  the  raters.  Include 
the  following  four  columns:  Data  Segment;  Theme;  Level;  Behavioral  Descriptor. 
Add  a  fifth  column  for  “Notes  or  Comments.” 

3 .  Print  out  the  four  T -BARS : 

Theme  1 :  Know  and  Use  All  Assets  Available 
Theme  2:  Focus  on  Mission  and  Higher’s  Intent 
Theme  3 :  Model  a  Thinking  Enemy/Populace 
Theme  4:  Consider  Effects  of  Terrain 

4.  Read  the  first  page  of  each  of  the  four  T-BARS  in  order  to  get  a  sense  of  what  the 
theme  is  about  and  what  each  of  the  five  levels  within  the  theme  is  intended  to 
represent  with  regard  to  performance  and  cognition.  Each  of  the  bullets  (marked  by  a 
letter  from  ‘A’  to  ‘M’)  within  a  column  describes  a  behavioral  indicator  that 
represents  cognitive  functioning  and  domain  mental  models  at  that  level  (1-5)  of 
performance. 

5.  Read  each  data  segment.  Select  the  theme  to  which  it  corresponds.  Then  within  the 
theme,  select  the  behavioral  descriptor  that  best  describes  the  data.  If  you  are  unable 
to  find  a  behavioral  descriptor  that  explicitly  describes  the  data,  then  consider  a) 
looking  at  another  theme,  or  b)  using  the  general  descriptors  of  each  level  within  the 
originally  selected  theme  to  rate  the  data.  Then  record  the  theme,  level,  and 
behavioral  descriptor  (bullet)  you’ve  selected  on  the  coding  sheet.  You  may  use  the 
context  provided  by  surrounding  data  to  code  a  particular  chunk  if  it  adds  clarity  to 
the  participant’s  response. 


2  For  information  on  the  interrater  reliability  testing  performed  by  the  T-BARS  developers,  see  Phillips,  Shafer, 
Ross,  Cox,  &  Shadrick  (2005). 
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6.  If  a  segment  seems  unrateable  because  it  lacks  the  content  required  to  make  sense,  or 
if  seems  to  be  an  aside  or  otherwise  unrelated  to  the  vignette  or  exercise,  then  do  not 
rate  it.  Simply  record  a  dash  in  that  cell  on  the  coding  sheet. 

7.  If  a  segment  seems  to  contain  elements  of  multiple  themes  or  multiple  levels,  then 
break  the  segment  apart  and  code  each  sub-segment.  Inform  the  other  rater(s)  of  the 
new  segment(s)  so  that  all  raters  code  identical  segments. 

8.  As  you  go  through  the  data,  record  any  issues  in  the  “Notes  or  Comments”  column  of 
the  coding  sheet.  For  example,  if  you  have  difficulty  discriminating  which  of  two  or 
three  behavioral  descriptors  best  fits  a  particular  data  segment,  record  the  options  you 
are  having  trouble  choosing  between.  If  you  find  any  of  the  behavioral  descriptors 
from  the  T-BARS  to  be  confusing,  record  those  issues  on  a  separate  sheet  of  paper. 

9.  Once  all  raters  have  completed  their  ratings,  compare  across  the  raters.  First  calculate 
percentage  agreement  across  the  Theme  ratings.  Then  calculate  percentage  agreement 
across  the  Level  ratings  ONLY  FOR  THE  SEGMENTS  ON  WHICH  RATERS 
AGREED  ON  THEME.  You  should  also  wish  to  calculate  the  number  of  cases  in 
which  raters’  agreement  on  Level  was  within  one  point  (e.g.,  one  rater  coded  a  ‘2’ 
and  another  coded  a  ‘1’  or  ‘3’). 

10.  Agreement  on  both  Themes  and  Levels  should  be  between  70%  and  90%  to  establish 
interrater  reliability.  If  your  agreement  is  less  than  70%,  look  at  the  cases  on  which 
the  raters  disagreed  and  discuss  each  rater’s  rationale.  You  will  need  to  decide  on 
your  own  rules  for  interpreting  the  behavioral  descriptors  when  there  is  not  immediate 
consensus.  We  recommend  documenting  these  rules  and  possibly  even  adding  them 
to  the  electronic  version  of  the  T-BARS. 

1 1.  Once  you  have  discussed  the  items  on  which  you  disagreed  and  calibrated  your 
interpretations  of  the  problematic  behavioral  descriptors,  repeat  the  process  for 
establishing  consensus  until  you  reach  at  least  70%  agreement. 

The  following  section  proved  the  actual  T-BARS  for  each  of  the  four  tactical  thinking  themes. 
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effects  of  using  specific  resources, 

assets  at  specific  times.  [Big 
Picture] 
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(Q)  Asks  questions  about  (Q)  Articulates  constraints 
availability  of  non-organic  due  to  troop  availability  vis- 
assets.  [Big  Picture  Thinking]  a-vis  mission. 
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(I)  Asks  questions  or  (I)  Describes  integrated 
identifies  information  picture  of  how  terrain  will 
needed  about  enemy  affect  asset(s)  or  mission, 
activity  along  key  terrain.  [Visualization] 
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(M)  Describes  route  features 
required  based  on  assets  used 
or  mission  tasks. 
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Appendix  A 

Sample  Tactical  Vignette  with  Verbal  Protocol 
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<Slide  1> 
NARRATOR: 


VIGNETTE  7:  Enable  Humanitarian  Operations 


The  Task  Force  Commander  has  assigned  your  team  to  escort  a  group  of  trucks  belonging 
to  several  non-governmental  and  private  volunteer  organizations  to  the  Khahaly  distribution 
center.  As  part  of  trying  to  reinforce  the  authority  of  the  central  government,  a  host  nation 
platoon  has  been  assigned  to  assist  in  the  escort  and  security  tasks.  Speed  is  important  for  this 
convoy  since  the  supplies  are  critically  needed.  Every  truck  is  also  vital  since  they  have  been 
loaded  with  single  commodities. 

< Slide  2> 

There  have  been  occasional  reports  of  paramilitary  groups  operating  illegal  checkpoints 
on  the  road.  In  the  last  week,  no  patrols  have  found  any  evidence  of  these  checkpoints  along 
Route  Relief. 

<Slide  3> 

PLATOON  LEADER: 

“Black  6,  this  is  White  1 .  The  bridge  up  here  is  washed  out  about  a  kilometer  past  CP  37. 
Looks  like  it’s  pretty  recent.  The  water  is  a  little  deep,  but  doesn’t  look  too  fast.  1  put  squads  out 
to  the  flanks  and  we  found  a  bypass  that  the  trucks  can  make.  Request  to  take  the  first  truck  in 
line  and  send  it  through?” 

Stopping  Point  #1 

■  What  are  the  critical  issues  or  facts  to  consider  at  this  point? 

■  I  know  you  don't  have  much  information  yet,  so  I'm  just  asking 
could  the  situation  unfold  with  what  you  know  now? 

■  What  would  be  your  biggest  concern  at  this  point?  What  would 

Lower  priority  questions: 

*  Have  you  seen  any  situation  like  this  before?  Briefly  describe. 

*  What  other  information  do  you  want  about  this  situation,  if  any?  How  would  you  get 
(element  of  information  mentioned)? 

■  How  would  that  information  affect  your  understanding  of  the  situation? 

NARRATOR: 

Passage  of  the  trucks  took  extra  time,  but  worked  out  without  any  becoming  stuck.  Just  as 
you  get  into  the  ceasefire  boundary  line  area,  you  come  across  a  large  washout  jammed  with 
trees  pulled  up  by  the  roots  that  has  choked  the  river  almost  closed.  The  riverbed  behind  is  filling 
with  water,  but  not  very  fast  -  the  road  is  still  high  and  dry  and  should  stay  so  for  at  least  a  day. 


for  speculation.  How 
be  your  top  priority? 


A -2 


You  continue  moving,  watching  the  river.  The  lead  platoon  leader  (White  1)  halts  the 
convoy  and  calls  you  forward.  As  you  crest  the  hill  where  the  platoon  leader  stopped,  you  see  a 
large  town  virtually  wiped  out  by  a  mining  catch  basin  that  has  broken  and  water  poured  through 
the  valley. 

On  entering  the  town  you  see  the  survivors  still  digging  out  of  the  water,  mud,  and  ruins. 
They  rush  to  the  trucks,  wailing  and  screaming,  thinking'the  supplies  are  for  them.  Many  of  the 
things  in  the  trucks  would  be  useful:  shovels,  blankets,  clothing,  food,  water,  basic  cooking 
items,  and  tents.  Some  of  the  NGO/PVO  truck  drivers  want  to  stop  here  and  help.  You  are  very 
conscious  of  the  suffering  the  supplies  are  meant  to  take  care  of  in  Khahlay  and  being  behind 
schedule. 

Stopping  Point  #2 

■  At  this  point,  what  are  the  critical  issues?  Has  this  caused  you  to  consider  any  other 
aspects  of  the  situation? 

■  (If  participant  tries  to  call  higher  for  guidance)  Would  you  be  making  a 
recommendation  to  higher? 

■  What  is  your  biggest  worry  or  concern? 

■  With  what  you  know,  how  do  you  think  this  situation  could  unfold? 

Lower  priority  questions: 

■  Have  you  seen  any  situation  like  this  before?  Briefly  describe. 

*  What  other  information  do  you  want  about  this  situation,  if  any?  How  would  you  get 
(element  of  information  mentioned)? 

■  How  would  that  information  affect  your  understanding  of  the  situation? 

<Slide  4> 

HOST  NATION  PLATOON  LEADER  (accented  English): 

“Not  stopping  here!  No  help!  We  GO  NOW!” 

VILLAGE  ELDER  (translated): 

“He  says:  Praise  the  all  Supreme  One  whose  name  we  are  not  fit  to  pronounce!  Our 
prayers  have  been  answered!  I  called  for  help  hours  ago  and  you  have  arrived!  What  have  you 
brought?  How  many  men?  We  can  still  save  some  of  the  people  who  were  buried  in  the  mud! 
This  way,  this  way!” 

NARRATOR: 

An  attempt  to  call  the  Task  Force  to  alert  them  to  the  crisis  here  is  unsuccessful  due  to  the 
steeply  sided,  narrow  valley  you  are  in.  It  would  take  over  20  minutes  to  reach  a  position  where 
the  radio  would  be  able  to  make  contact. 
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Stopping  Point  #3 


•  At  this  point,  what  are  you  considering? 

•  What  are  you  most  concerned  about?  Why? 

•  How  will  this  situation  unfold?  Why? 

•  Describe  the  rationale  behind  your  course  of  action. 

•  With  whom  will  you  communicate  and  why? 


A -4 


